Papers with real-world applications
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track (2025.emnlp-industry)
Copied to clipboard
| Challenge: | EMNLP 2025 Industry Track highlights key insights, novel research trends and challenges encountered in practical language technology applications. |
| Approach: | Kai Chen will present the technical advances behind the open-source Intern-series large models . he will highlight how models acquire expert-level skills in specialized domains . |
| Outcome: | This talk will highlight the technical advances behind the open-source Intern-series models . it will highlight how models acquire expert-level skills in specialized domains while retaining broad generalization ability. |
Pretrained Transformers for Text Ranking: BERT and Beyond (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | This tutorial provides an overview of text ranking using neural network architectures known as transformers. |
| Approach: | This tutorial provides an overview of text ranking with neural network architectures known as transformers. |
| Outcome: | This tutorial provides an overview of text ranking with neural network architectures known as transformers. |
Meaning Representations for Natural Languages: Design, Models and Applications (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models. |
| Approach: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models. |
| Outcome: | This tutorial reviews the design of common meaning representations and SoTA models for predicting meaning representation models . it also reviews the applications of meaning representation in downstream NLP tasks and real-world applications . |
Tutorial: End-to-End Speech Translation (2021.eacl-tutorials)
Copied to clipboard
| Challenge: | Speech translation is the translation of speech in one language typically to text in another, traditionally accomplished through a combination of automatic speech recognition and machine translation. |
| Approach: | This tutorial introduces the techniques used in cutting-edge research on speech translation. |
| Outcome: | The proposed models achieve state-of-the-art performance with end-to-end speech translation for both high- and low-resource languages. |
Tutorial on Multimodal Machine Learning (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Multimodal machine learning is a challenging but crucial area with numerous applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. |
| Approach: | This tutorial will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research. |
| Outcome: | The proposed taxonomy synthesizes the core technical challenges and major directions for future research. |
Representation, Learning and Reasoning on Spatial Language for Downstream NLP Tasks (2020.emnlp-tutorials)
Copied to clipboard
| Challenge: | In this tutorial, we discuss the cutting-edge research results and existing challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Approach: | This tutorial presents cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
| Outcome: | This paper reviews the cutting-edge research results and current challenges related to spatial language understanding including semantic annotations, existing corpora, symbolic and sub-symbolic representations, qualitative spatial reasoning, spatial common sense, deep and structured learning models. |
ResPer: Computationally Modelling Resisting Strategies in Persuasive Conversations (2021.eacl-main)
Copied to clipboard
Ritam Dutt, Sayan Sinha, Rishabh Joshi, Surya Shekhar Chakraborty, Meredith Riggs, Xinru Yan, Haogang Bao, Carolyn Rose
| Challenge: | Existing research has failed to account for resisting strategies employed to foil persuasion attempts. |
| Approach: | They propose a framework for identifying resisting strategies in persuasive conversations . they instantiate a dataset comprising persuasion and negotiation conversations based on a hierarchical sequence-labelling neural architecture . |
| Outcome: | The proposed framework is based on two persuasive conversation datasets and leverages a hierarchical sequence-labelling neural architecture to infer resisting strategies automatically. |
CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databases (2025.naacl-long)
Copied to clipboard
Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Qizhe Shieh, Wenmeng Zhou
| Challenge: | Large Language Models excel in stand-alone code tasks but struggle with handling entire code repositories. |
| Approach: | They propose a system that integrates LLM agents with graph database interfaces extracted from code repositories. |
| Outcome: | The proposed system integrates LLM agents with graph database interfaces extracted from code repositories. |
In-BoXBART: Get Instructions into Biomedical Multi-Task Learning (2022.findings-naacl)
Copied to clipboard
| Challenge: | Experimental results show that the proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average. |
| Approach: | They propose a unified model that can learn all 32 instruction tasks of the BoX without any task-specific modules. |
| Outcome: | The proposed model outperforms single-task baseline by 3% and multi-task (without instruction) baseline by 18% on an average. |
RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks (2025.emnlp-industry)
Copied to clipboard
Amit Agarwal, Hitesh Laxmichand Patel, Srikant Panda, Hansa Meghwani, Jyotika Singh, Karan Dua, Paul Li, Tao Sheng, Sujith Ravi, Dan Roth
| Challenge: | Existing evaluation methods do not explicitly measure this distinction, hindering effective dataset curation and real-world focused model development. |
| Approach: | They introduce a region-based score to quantify a dataset's reliance on global versus local visual information. |
| Outcome: | The proposed model-based score systematically compares model performance on image patches versus full images to determine if tasks require holistic image understanding or can be solved with partial or localized visual cues. |
Benchmarking LLM’s Capability in Reasoning over Conflicting Web References (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) integrated with retrieval-augmented generation (RAG) are a dominant framework for building intelligent assistants. |
| Approach: | They propose a benchmark to evaluate LLMs' reasoning capability over real-world conflicting documents retrieved from the web. |
| Outcome: | The proposed benchmark evaluates LLMs' reasoning capability over real-world conflicting documents retrieved from the web. |
A Diverse and Effective Retrieval-Based Debt Collection System with Expert Knowledge (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing debt collection systems lack script diversity, contextual relevance and coherence due to their complexity. |
| Approach: | They propose a script library based on real-world debt collection conversations and a retrieval based response system for contextual relevance. |
| Outcome: | The proposed system improves script diversity and responds to debtor-collector conversations better through knowledge distillation. |
Inference to the Best Explanation in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have found success in real-world applications, but their underlying explanatory process is still poorly understood. |
| Approach: | They propose to use a framework inspired by philosophical accounts on Inference to the Best Explanation (IBE) to advance the interpretation and evaluation of LLMs’ explanations. |
| Outcome: | The proposed framework can identify the best explanation with up to 77% accuracy (27% above random) while being intrinsically more efficient and interpretable. |
Squrve: A Unified and Modular Framework for Complex Real-World Text-to-SQL Tasks (2026.acl-demo)
Copied to clipboard
| Challenge: | Existing methods are designed for specific settings, each with its own set of challenges. |
| Approach: | They propose a unified, modular, and extensive Text-to-SQL framework . it proposes a universal execution paradigm and a multi-actor collaboration mechanism . |
| Outcome: | Squrve proposes a unified, modular, and extensive Text-to-SQL framework . the framework outperforms existing methods on widely adopted benchmarks . |
Learning When Not to Answer: a Ternary Reward Structure for Reinforcement Learning Based Question Answering (N19-2)
Copied to clipboard
| Challenge: | Existing methods for question answering over knowledge graphs use reinforcement learning to reason over a knowledge graph. |
| Approach: | They propose a new performance metric for question-answering agents that extends the binary reward structure to a ternary reward structure which rewards an agent for not answering a question rather than giving an incorrect answer. |
| Outcome: | The proposed method significantly improves the precision of answered questions while only not answering a limited number of correctly answered questions. |
Thesis Proposal: On the Granularity-Robustness Trade-off in Text-Derived Knowledge Graphs (2026.acl-srw)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) based on dense embeddings is a dominant paradigm for text retrieval, but many real-world applications require attribute-specific querying. |
| Approach: | They propose a query-driven framework for constructing and retrieving knowledge graphs from text using dense embeddings. |
| Outcome: | The proposed framework combines the robustness of dense retrieval with the explicit queryability of symbolic representations. |
BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases (2023.acl-short)
Copied to clipboard
| Challenge: | Energy-based models (EBMs) have gained popularity for controlled text generation due to their high applicability to a wide range of constraints. |
| Approach: | They propose a language model with tunable biases to adjust the language model’s output logits. |
| Outcome: | The proposed model maintains the generator’s autoregressive nature to assert a strong control on token-wise conditional dependencies and overall fluency, and converges faster. |
SMARTAVE: Structured Multimodal Transformer for Product Attribute Value Extraction (2022.findings-emnlp)
Copied to clipboard
Qifan Wang, Li Yang, Jingang Wang, Jitin Krishnan, Bo Dai, Sinong Wang, Zenglin Xu, Madian Khabsa, Hao Ma
| Challenge: | Existing methods for product attribute value extraction are noisy and incomplete with missing values for most retailers. |
| Approach: | They propose a Structure Mltimodal trAnsformeR for producT Attribute Value Extraction which jointly encodes the structured product information from multiple modalities. |
| Outcome: | The proposed method outperforms state-of-the-art methods on two multimodal product datasets. |
Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to extract relationships from texts depend on memory size and replay these memorized samples in subsequent tasks. |
| Approach: | They propose to use a model to extract relations between entities from texts where the samples of different relations are delivered into the model continuously. |
| Outcome: | The proposed model outperforms the state-of-the-art models and avoids catastrophic forgetting. |
DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) with web search capabilities show significant potential for deep research. |
| Approach: | They introduce a framework for end-to-end training of LLM-based deep research agents . they implement a specialized multi-agent architecture where browsing agents extract relevant information from various webpage structures. |
| Outcome: | The proposed framework improves on open-domain research tasks by 28.9 points over prompt engineering and 7.2 points over RAG-based RL agents. |
Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond (2025.acl-industry)
Copied to clipboard
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Tanglifu Tanglifu, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang
| Challenge: | Experimental results show that opensource curriculum training is more effective when distinct datasets are available for different training stages. |
| Approach: | They propose an opensource suite for training long reasoning models using publicdata and models. |
| Outcome: | The proposed model outperforms DeepSeek-R1-DistillQwen-32B models in math reasoning. |
Grafting Pre-trained Models for Multimodal Headline Generation (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing approaches to generate video headlines with pre-trained language models are labor intensive and impractical. |
| Approach: | They propose to graft the encoder from the pre-trained video-language model on the generative pre-trainer model and propose a consensus fusion mechanism for the integration of different components. |
| Outcome: | The proposed model achieves strong results on a brand-new dataset collected from real-world applications. |
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing benchmarks for long-context LLMs focus on generic tasks that are not necessarily aligned with real-world applications. |
| Approach: | They propose to augment existing ELITR corpus by adding 271 manually crafted questions with their ground-truth answers and noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
| Outcome: | The proposed benchmark augments the existing ELITR corpus by adding 271 manually crafted questions with ground-truth answers, as well as noisy versions of meeting transcripts altered to target different Word Error Rate levels. |
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets (2023.findings-acl)
Copied to clipboard
Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, Jimmy Huang
| Challenge: | Currently, the evaluation of large language models (LLMs) such as ChatGPT in academic datasets is difficult due to the difficulty of evaluating the generative outputs produced by this model against the ground truth. |
| Approach: | They evaluate ChatGPT across 140 tasks and analyze 255K responses it generates in academic datasets. |
| Outcome: | The proposed model performs well on 140 tasks and generates 255K responses in these datasets. |
Large Scale Generative Multimodal Attribute Extraction for E-commerce Attributes (2023.acl-industry)
Copied to clipboard
| Challenge: | E-commerce websites often don’t label or mislabel attributes of products . |
| Approach: | They propose a multi-modal product attribute generation system that extracts product attributes from the product pages of eCommerce stores by using both text and images. |
| Outcome: | The proposed model improves the recall@90P accuracy by 10.16% and 6.9 from the state-of-the-art models. |
Emotion Transfer with Enhanced Prototype for Unseen Emotion Recognition in Conversation (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on emotion recognition in conversation does not reach a consensus on classification theories . despite this, there is no clear consensus on how to recognize previously unseen emotions in real-world applications. |
| Approach: | They propose a prototype-based emotion transfer framework that can be used in real-world applications. |
| Outcome: | The proposed framework shows promise but still faces key challenges in the field of emotion recognition in conversation. |
Query-Efficient Textual Adversarial Example Generation for Black-Box Attacks (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing black-box attacks require thousands of queries on the target model, making them expensive in real-world applications. |
| Approach: | They propose a new approach that guides word substitutions using prior knowledge from the training set to improve the attack efficiency. |
| Outcome: | The proposed approach reduces query-free attack and guided search attacks by a factor of 10 500 . it improves transferability and generalization by the ensemble of the ABPens in NLP . |
DateLogicQA: Benchmarking Temporal Biases in Large Language Models (2025.naacl-srw)
Copied to clipboard
| Challenge: | DateLogicQA examines temporal biases in Large Language Models (LLMs) 190 questions are curated by humans to examine temporal reasoning across date formats and contexts . |
| Approach: | They propose a human-curated benchmark of 190 questions specifically designed to understand temporal bias in Large Language Models. |
| Outcome: | The proposed dataset covers seven date formats across past, present, and future contexts . it examines four reasoning types: commonsense, factual, conceptual, and numerical . |
Improving Text-to-SQL Evaluation Methodology (P18-1)
Copied to clipboard
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, Dragomir Radev
| Challenge: | Current evaluations of text-to-SQL systems are limited by the way they divide data into training and test sets. |
| Approach: | They propose to standardize and improve existing and new text-to-SQL datasets . they propose a template-based slot-filling baseline that cannot generalize to new queries . |
| Outcome: | The proposed system is competitive with prior work on multiple datasets and can be used on training and test sets. |
MonoTODia: Translating Monologue Requests to Task-Oriented Dialogues (2025.naacl-industry)
Copied to clipboard
| Challenge: | Data scarcity is one of the main problems when it comes to real-world applications of transformer-based models. |
| Approach: | They propose a method to source annotated German monologues from existing monologue material to train TOD systems. |
| Outcome: | The proposed model can be used to train TOD systems on a real-world example of a travel booking service. |
Zero-Shot Strategies for Length-Controllable Summarization (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large language models struggle with precise length control, particularly in zero-shot settings. |
| Approach: | They propose to use length approximation, target adjustment, sample filtering and automated revisions to improve LLMs' length control capabilities. |
| Outcome: | The proposed methods improve length control in large language models while maintaining or enhancing summary quality without the need for model fine-tuning or architectural changes. |
Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce (P18-2)
Copied to clipboard
| Challenge: | Recent researches focus on deep learning and reinforcement learning for multi-turn information seeking conversation systems. |
| Approach: | They propose an efficient and effective multi-turn conversation model based on convolutional neural networks and extend it to adapt the knowledge learned from a resource-rich domain to enhance the performance. |
| Outcome: | The proposed model performs better than the existing model on an industrial chatbot called AliMe Assist. |
SFAL: Semantic-Functional Alignment Scores for Distributional Evaluation of Auto-Interpretability in Sparse Autoencoders (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Interpreting the internal representations of large language models (LLMs) is crucial for their deployment in real-world applications, impacting areas such as AI safety, debugging, and compliance. |
| Approach: | They propose an alternative evaluation strategy that assesses the alignment between the semantic neighbourhoods of features and their functional neighbourhoods by using co-occurrence statistics. |
| Outcome: | The proposed evaluation strategy reduces reliance on scoring on large-scale models and improves efficiency and cost-effectiveness. |
SCORE: Systematic COnsistency and Robustness Evaluation for Large Language Models (2025.naacl-industry)
Copied to clipboard
| Challenge: | Typical evaluations of Large Language Models (LLMs) report a single accuracy metric per dataset, often derived from an optimized setup. |
| Approach: | They propose a framework for non-adversarial evaluation of large language models that evaluates models by repeatedly testing them on the same benchmarks in various setups. |
| Outcome: | The proposed framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. |
TestAgent: An Adaptive and Intelligent Expert for Human Assessment (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing adaptive testing methods face several challenges due to mechanized nature of most algorithms and noisy response data. |
| Approach: | They propose to use large language models to enhance adaptive testing through interactive engagement to capture test-takers’ responses and anomalies. |
| Outcome: | The proposed agent achieves more accurate results with 20% fewer questions than state-of-the-art baselines and testers preferred it in speed, smoothness, and other dimensions. |
FAST: Fast Annotation tool for SmarT devices (2021.emnlp-demo)
Copied to clipboard
| Challenge: | In real-world applications, annotators with the same attributes are required to annotate whether the outputs of natural language generation systems are fluent or not. |
| Approach: | They propose an annotation tool for application tasks that focuses on the user experience of mobile devices and can be customized to fit various tasks. |
| Outcome: | The proposed tool can annotate faster than existing methods while maintaining the quality of annotation. |
Deploying Multi-task Online Server with Large Language Model (2025.coling-industry)
Copied to clipboard
| Challenge: | In the industry, numerous natural language processing tasks are deployed online . traditional approaches tackle each task separately by its own network and pipeline . |
| Approach: | They propose a three-stage multi-task learning framework for large language models . it involves task filtering, fine-tuning on high-resource tasks, and finally fine- tuning on all tasks . |
| Outcome: | The proposed framework reduces up to 90% of overhead while reducing latency and resource usage. |
Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated Data (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing text classification methods focus on a fixed label set, but many real-world applications require extending to new fine-grained classes as the number of samples per label increases. |
| Approach: | They propose a problem called coarse-to-fine grained classification that leverages label surface names as the only human guidance. |
| Outcome: | The proposed method outperforms existing methods on two real-world datasets. |
OpenBioNER: Lightweight Open-Domain Biomedical Named Entity Recognition Through Entity Type Description (2025.findings-naacl)
Copied to clipboard
Alessio Cocchieri, Giacomo Frisoni, Marcos Martínez Galindo, Gianluca Moro, Giuseppe Tagliavini, Francesco Candoli
| Challenge: | Biomedical Named Entity Recognition (BioNER) is a computationally expensive and limited tool . specialized 7B NER LLMs and GPT-4o can't match textual spans with entity types . |
| Approach: | They propose a lightweight BERT-based cross-encoder architecture that can identify any biomedical entity using only its description. |
| Outcome: | The proposed system outperforms existing models that match textual spans with entity types rather than descriptions on biomedical benchmarks. |
Fast Nearest Neighbor Machine Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Fast kNN-MT uses the entire corpus as the datastore for the nearest neighbor search . knn-MT is two-orders slower than vanilla MT models . |
| Approach: | They propose a fast kNN-MT model that uses the entire corpus as the datastore for nearest neighbor search. |
| Outcome: | The proposed model is two-orders faster than kNN-MT and is only two times slower than the standard model. |
Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA (2023.acl-long)
Copied to clipboard
| Challenge: | Despite remarkable progress made in natural language processing, even the state-of-the-art systems often make incorrect predictions. |
| Approach: | They propose to use selective prediction to enable models to abstain from answering when their predictions are likely to be incorrect. |
| Outcome: | The proposed method improves performance on 11 QA datasets and in- and out-of-domain settings. |
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus (2023.emnlp-main)
Copied to clipboard
Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, Luoyi Fu
| Challenge: | Existing methods for detecting hallucinations in LLMs rely on external knowledge for reference retrieval or require sampling multiple responses for consistency verification. |
| Approach: | They propose a reference-free, uncertainty-based method for detecting hallucinations in Large Language Models that imitates human focus in factuality checking from three aspects: focus on the most informative keywords; focus on unreliable tokens in historical context; focus of token properties such as token type and token frequency. |
| Outcome: | The proposed method achieves state-of-the-art performance across all evaluation metrics and eliminates the need for additional information. |
ObfusLM: Privacy-preserving Language Model Service against Embedding Inversion Attacks (2025.acl-long)
Copied to clipboard
Yu Lin, Ruining Yang, Yunlong Mao, Qizhi Zhang, Jue Hong, Quanwei Cai, Ye Wu, Huiqi Liu, Zhiyu Chen, Bing Duan, Sheng Zhong
| Challenge: | Recent studies show that obfuscation techniques for MLaaS are susceptible to embedding inversion attacks (EIAs). |
| Approach: | They propose a model obfuscation framework that protects client inputs from embedding inversion attacks by obliviously obbing models. |
| Outcome: | The proposed framework outperforms existing works in utility by 10% with a nearly 80% resistance rate against embedding inversion attacks. |
How to tackle an emerging topic? Combining strong and weak labels for Covid news NER (2022.aacl-short)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) models can be trained for emerging topics such as medical domain where new topics are constantly evolving out of the scope of existing models and datasets. |
| Approach: | They propose a recipe to combine weak and strong labels to improve Named Entity Recognition (NER) models for emerging topics. |
| Outcome: | The proposed model outperforms methods trained on weak data while combining out-of-domain and in-domain weak label training. |
Transparent Reference-free Automated Evaluation of Open-Ended User Survey Responses (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to evaluate open-ended survey responses are expensive and lack ground-truth reference for comparison. |
| Approach: | They propose a two-stage evaluation framework specifically designed for human survey responses that uses gibberish filtering to remove nonsensical responses. |
| Outcome: | The proposed evaluation framework outperforms existing metrics on English and Korean datasets and shows strong correlations with expert assessment. |
RAD-Bench: Evaluating Large Language Models’ Capabilities in Retrieval Augmented Dialogues (2025.naacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks assess LLMs' chat abilities in multi-turn dialogues or their use of retrieval for augmented responses in limited tasks such as knowledge QA or numeric reasoning. |
| Approach: | They propose a benchmark to evaluate LLMs' capabilities in multi-turn dialogues following retrievals. |
| Outcome: | The proposed benchmark evaluates LLMs' ability to perform in multi-turn dialogues following retrievals over 6 representative scenarios. |
Unsupervised Single Document Abstractive Summarization using Semantic Units (2022.aacl-main)
Copied to clipboard
| Challenge: | a lack of sufficient training pairs is a common issue in real-world applications. |
| Approach: | They propose a framework that lets a model learn the frequency of each semantic unit in the source text. |
| Outcome: | The proposed model outperforms other unsupervised methods under CNN/Daily Mail task. |
ThinkBooster: A Unified Framework for Seamless Test-Time Scaling of LLM Reasoning (2026.acl-demo)
Copied to clipboard
Vladislav Smirnov, Quang-Chieu Nguyen, Sergey Senichev, Minh Ngoc Ta, Ekaterina Fadeeva, Artem Vazhentsev, Daria Galimzianova, Nikolai Rozanov, Viktor Mazanov, Jingwei Ni, Tianyi Wu, Igor Kiselev, Mrinmaya Sachan, Iryna Gurevych, Preslav Nakov, Timothy Baldwin, Artem Shelmanov
| Challenge: | Existing TTC scaling strategies and reasoning scorers are fragmented and evaluated under inconsistent protocols. |
| Approach: | They propose a framework for seamless test-time compute scaling of large language model reasoning . they use a modular Python library to implement state-of-the-art scaling strategy and scorer families . |
| Outcome: | The proposed framework evaluates performance and computational efficiency on mathematical and coding tasks. |
CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation (2025.findings-naacl)
Copied to clipboard
Yiruo Cheng, Kelong Mao, Ziliang Zhao, Guanting Dong, Hongjin Qian, Yongkang Wu, Tetsuya Sakai, Ji-Rong Wen, Zhicheng Dou
| Challenge: | Existing research focuses on single-turn RAG, leaving a gap in addressing multi-turn conversations . a new benchmark is designed to assess RAG systems in realistic multi-turned conversations based on Wikipedia . |
| Approach: | They propose a large-scale benchmark to assess RAG systems in multi-turn contexts . CORAL includes diverse information-seeking conversations automatically derived from Wikipedia . authors propose unified framework to standardize various conversational RAG methods . |
| Outcome: | The proposed framework supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
Hard Negative Mining for Domain-Specific Retrieval in Enterprise Systems (2025.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for lexical retrieval struggle due to semantic mismatches and overlapping terminologies, and ambiguous abbreviations common in specialized fields like finance and cloud computing. |
| Approach: | They propose a scalable hard-negative mining framework that dynamically selects semantically challenging but contextually irrelevant documents to enhance deployed re-ranking models. |
| Outcome: | The proposed framework improves on public domain datasets and shows that it is generalizable and ready for real-world applications. |
KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models (2023.acl-industry)
Copied to clipboard
| Challenge: | Image ad understanding is a crucial task with wide real-world applications, but is under-explored in the machine learning community due to the lack of foundational vision-language models (VLMs) . |
| Approach: | They propose a simple feature adaptation strategy to fuse multimodal information for image ads and further empower it with knowledge of real-world entities. |
| Outcome: | The proposed strategy fuses multimodal information for image ads and empowers it with knowledge of real-world entities. |
Few-Shot Table-to-Text Generation with Prototype Memory (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Neural table-to-text generation models are data-hungry and require large amounts of training data to learn the mapping between tables and texts. |
| Approach: | They propose a framework for table-to-text generation under the few-shot scenario that uses retrieved prototypes and a prototype selector to bridge the structural gap between tables and texts. |
| Outcome: | The proposed framework significantly improves the model performance on three benchmark datasets with state-of-the-art models. |
Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal (2024.acl-long)
Copied to clipboard
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su
| Challenge: | Existing methods to train LLMs on previous training data are not feasible in real-world applications because of catastrophic forgetting. |
| Approach: | They propose a framework that uses the LLM to generate synthetic instances for rehearsal and refine the instance outputs based on the synthetic inputs. |
| Outcome: | The proposed framework achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. |
Language, OCR, Form Independent (LOFI) pipeline for Industrial Document Information Extraction (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models for low-resource language (LRL) documents are limited in semantic entity extraction, and there are limitations in SER from word level results. |
| Approach: | They propose a pipeline for Document Information Extraction (DIE) in low-resource language (LRL) business documents that solves language, Optical Character Recognition (OCR), and form dependencies through flexible model architecture, a token-level box split algorithm, and the SPADE decoder. |
| Outcome: | Experiments on Korean and Japanese documents show that the pipeline performs well in the Semantic Entity Recognition task without pre-training. |
SelfMix: Robust Learning against Textual Label Noise with Self-Mixup Training (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to handle label noise in text classification tasks are limited to visual data. |
| Approach: | They propose a method to handle label noise in text classification tasks using a Gaussian Mixture Model. |
| Outcome: | The proposed method outperforms baselines on three types of text classification tasks on visual and textual data. |
On the Impacts of Contexts on Repository-Level Code Generation (2025.findings-naacl)
Copied to clipboard
| Challenge: | CodeLLMs are widely used for code generation, but their ability to handle repository-level dependencies remains underexplored. |
| Approach: | They propose a benchmark for evaluating repository-level code generation based on dependency contexts. |
| Outcome: | The proposed model improves dependency handling and introduces a new metric, Dependency Invocation Rate (DIR), to measure context utilization. |
CRISP: Persistent Concept Unlearning via Sparse Autoencoders (2026.acl-long)
Copied to clipboard
| Challenge: | Recent work has explored sparse autoencoders (SAEs) to perform precise interventions on monosemantic features, but most SAE-based methods operate at inference time, which does not create persistent changes in the model’s parameters. |
| Approach: | They propose a parameter-efficient method for persistent concept unlearning using SAEs that automatically identifies salient SAE features across multiple layers and suppresses their activations. |
| Outcome: | The proposed method outperforms previous methods on safety-critical unlearning tasks from the WMDP benchmark, successfully removing harmful knowledge while preserving general and in-domain capabilities. |
E-VarM: Enhanced Variational Word Masks to Improve the Interpretability of Text Classification Models (2022.coling-1)
Copied to clipboard
Ling Ge, ChunMing Hu, Guanghui Ma, Junshuang Wu, Junfan Chen, JiHong Liu, Hong Zhang, Wenyi Qin, Richong Zhang
| Challenge: | Empirical studies show that our approach outperforms the SOTA methods in improving the interpretability of text classification models. |
| Approach: | They propose an enhanced variational word masks approach that exploits the Variational Information Bottleneck to obtain task-specific words. |
| Outcome: | Empirical results show that the proposed method outperforms the SOTA methods in improving the interpretability of the model. |
Certified Robustness to Word Substitution Attack with Differential Privacy (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies have shown that adversarial examples can be easily fooled by DNNs, making the robustness and security of NLP models significantly important. |
| Approach: | They propose a differential privacy-based algorithm to achieve certified robustness against word substitution at- tacks in text classification via differential privacy. |
| Outcome: | The proposed model achieves higher accuracy and more than 30X efficiency improvement over existing defense algorithms. |
A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) can disentangle complex features into more interpretable components. |
| Approach: | They propose to use Sparse Autoencoders to disentangle LLM features into more interpretable components. |
| Outcome: | The proposed method disentangles complex features into more interpretable components. |
Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series (2025.emnlp-industry)
Copied to clipboard
| Challenge: | In the rapidly evolving landscape of large language models, the need for efficient reasoning models has become increasingly urgent. |
| Approach: | They extend the Qwen model family by introducing four model series specifically designed for industrial applications. |
| Outcome: | The proposed models outperform previous models in multiple benchmarks and provide scalable training and inference functionality on the Alibaba Cloud PAI platform. |
Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for multimodal sentiment analysis often fail due to equipment failure, data corruption, privacy issues and the like. |
| Approach: | They propose a multimodal Transformer framework using prompt learning to address the issue of missing modalities. |
| Outcome: | The proposed framework outperforms existing methods significantly across evaluation metrics. |
Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing studies on robustness of pretrained multilingual models are limited to the English language. |
| Approach: | They propose to use data augmentation and contrastive loss term to boost robustness of multilingual models in cross-lingual settings. |
| Outcome: | The proposed model outperforms existing models on clean and noisy data in the cross-lingual setting. |
Abstract Meaning Representation for Multi-Document Summarization (C18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic representation of natural language based on linguistic theory . |
| Approach: | They propose to use Abstract Meaning Representation (AMR) as a content representation. |
| Outcome: | The proposed framework is fully data-driven and flexible. |
SCCS: Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment (2023.findings-acl)
Copied to clipboard
Jielin Qiu, Jiacheng Zhu, Mengdi Xu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Bo Li, Ding Zhao, Hailin Jin
| Challenge: | Existing methods for multimodal summarization ignore the structure and semantics of the whole video and article. |
| Approach: | They propose a semantic-consistent cross-domain summarization model that extracts features from video and article and uses fusion methods to select representative one. |
| Outcome: | The proposed model produces high-quality multimodal summaries on three MSMO datasets. |
A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphs (2025.acl-industry)
Copied to clipboard
Chiwan Park, Wonjun Jang, Daeryong Kim, Aelim Ahn, Kichang Yang, Woosung Hwang, Jihyeon Roh, Hyerin Park, Hyosun Wang, Min Seok Kim, Jihoon Kang
| Challenge: | Large Language Models (LLMs) have led to significant improvements in various service domains, including search, recommendation, and chatbot applications. |
| Approach: | They propose a framework for developing scalable, controllable, and reliable AI-driven agents that can be applied to real-world applications. |
| Outcome: | The proposed framework bridges the gap between academic research and real-world application, and enables scalable, controllable, and reliable AI-driven agents. |
SMART: Scalable Multilingual Approach for a Robust TOD System (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing TOD frameworks face significant challenges in handling unstructured information, providing multilingual support, and engaging proactively. |
| Approach: | They propose a novel TOD framework that combines traditional pipeline elements with modern agent-based approaches and features a simplified dialogue state, intelligent clarification mechanisms, and a unified natural language generation component that eliminates response redundancy. |
| Outcome: | The proposed framework outperforms baseline systems across key metrics and integrates in an e-commerce store. |
Evaluating the Performance of Large Language Models via Debates (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are evolving and impacting various fields . current methods for evaluation are based on fixed, domain-specific questions or rely on human input, making them unscalable. |
| Approach: | They propose a benchmarking framework based on debates between LLMs, judged by another LLM. |
| Outcome: | The proposed framework achieves rankings that align closely with popular rankings based on human input eliminating the need for costly crowdsourcing. |
Sentiment Word Aware Multimodal Refinement for Multimodal Sentiment Analysis with ASR Errors (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing models for multimodal sentiment analysis are limited in their capacity to be deployed in the real world. |
| Approach: | They propose a model that can dynamically refine erroneous sentiment words by leveraging multimodal sentiment clues. |
| Outcome: | The proposed model surpasses the state-of-the-art models on three datasets. |
Reinforcement Learning with Token-level Feedback for Controllable Text Generation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for controllable text generation are guided by coarse-grained feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. |
| Approach: | They propose a reinforcement learning algorithm which formulates TOken-LEvel rewards for controllable text generation and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm. |
| Outcome: | The proposed algorithm can achieve superior performance on single-attribute and multi-attract control tasks. |
Mixture-of-Minds: Multi-Agent Reinforcement Learning for Table Understanding (2026.acl-long)
Copied to clipboard
Yuhang Zhou, Mingrui Zhang, Ke Li, Mingyi Wang, Qiao Liu, Qifei Wang, Jiayi Liu, Fei Liu, Serena Li, Weiwei LI, Mingze Gao, Abhishek Kumar, Xiangjun Fan, Zhuokai Zhao, Lizhu Zhang
| Challenge: | Large language models (LLMs) have shown promise on understanding and reasoning over tables, but current approaches remain limited. |
| Approach: | They propose a multi-agent framework that decomposes table reasoning into three specialized roles: planning, coding, and answering. |
| Outcome: | The proposed framework decomposes table reasoning into three specialized roles: planning, coding, and answering. |
DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications (2021.acl-short)
Copied to clipboard
| Challenge: | In order to comprehensively verify the robustness and generalization of MRC models, we construct a real-world Chinese dataset - DuReader_robust . |
| Approach: | They introduce a real-world Chinese dataset to evaluate the robustness and generalization of MRC models from three aspects: over-sensitivity, over-stability and generalisation. |
| Outcome: | The proposed model fails to perform well on the challenge test set and may provide suggestions for future model development. |
Rethinking Scale: Deployment Trade-offs of Small Language Models under Agent Paradigms (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing research focuses on enhancing large language models through scaling laws or fine-tuning strategies, but ignores the potential of using agent paradigms to compensate for the inherent weaknesses of small models. |
| Approach: | They propose to use structured agent frameworks to improve effectiveness over direct prompting . they also propose to employ routing-based multi-agent systems with collaborative capabilities . |
| Outcome: | The proposed model significantly outperforms direct prompting with single-agent systems . the proposed model is more reliable and cost-effective than other models . |
Disentangling Language and Knowledge in Task-Oriented Dialogs (N19-1)
Copied to clipboard
| Challenge: | Existing approaches to handle task-oriented dialogs break when asked to handle such changes. |
| Approach: | They propose an encoder-decoder architecture with a novel Bag-of-Sequences memory which facilitates the disentangled learning of the response’s language model and its knowledge incorporation. |
| Outcome: | The proposed architecture outperforms state-of-the-art models on bAbI OOV test sets and other human-human datasets and shows that it is robust to KB modifications. |
Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification require task-specific training data in high-resource sources . labeling cost, task characteristics, and privacy concerns can hinder the use of cross-linguistic training . |
| Approach: | They propose a dictionary-based heterogeneous graph (DHGNet) that uses bilingual dictionaries for task-independent word embeddings. |
| Outcome: | The proposed method outperforms pretrained models even though it does not access to large corpora. |
RealTalk-CN: A Realistic Chinese Speech Task-Oriented Dialogue Benchmark with Cross-Modal Analysis (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in speech large language models have enabled end-to-end spoken interactions, but their robustness in real-world applications remains unclear. |
| Approach: | They propose a multi-turn, multi-domain speech–text TOD dataset for Chinese users . it contains 5.4k dialogues with annotations for dialogue states, disfluency types, speaker characteristics . |
| Outcome: | The proposed model can be used to evaluate speech large language models in real-world scenarios . the proposed model is based on 5.4k real human-to-human dialogues with annotations . |
Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Framework (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on retrieval-augmented generation (RAG) rarely address the issue of predictive uncertainty, i.e., how likely it is that a RAG model’s prediction is incorrect. |
| Approach: | They propose a framework that induces RAG models to alter latent factors and analyzes the effect on their answers. |
| Outcome: | The proposed framework identifies two critical factors affecting RAG models' confidence in their answers and analyzes the effect on their answers. |
Can Large Language Models Understand Context? (2024.findings-eacl)
Copied to clipboard
Yilun Zhu, Joel Moniz, Shruti Bhargava, Jiarui Lu, Dhivya Piraviperumal, Site Li, Yuan Zhang, Hong Yu, Bo-Hsiang Tseng
| Challenge: | Existing evaluation methodologies for Large Language Models (LLMs) have been inadequate to evaluate their ability to understand contextual features. |
| Approach: | They propose a benchmark to assess large language models' ability to understand context by adapting existing datasets to suit their evaluation. |
| Outcome: | The proposed model performs better under the in-context learning pretraining scenario than state-of-the-art models. |
CMQA: A Dataset of Conditional Question Answering with Multiple-Span Answers (2022.coling-1)
Copied to clipboard
| Challenge: | Existing QA datasets only contain unconditional and parallel answers . conditional question answering with hierarchical multi-span answers is challenging for the community to solve . |
| Approach: | They propose a conditional question answering task with hierarchical multi-span answers . they propose CMQA, which contains conditional and hierarchic samples . |
| Outcome: | The proposed task can be used to build more reliable and sophisticated QA systems. |
METAL: Towards Multilingual Meta-Evaluation (2024.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that Large Language Models excel on many standard NLP benchmarks. |
| Approach: | They propose a framework for end-to-end evaluation of Large Language Models as evaluators in multilingual scenarios. |
| Outcome: | The proposed framework evaluates LLMs as evaluators in multilingual scenarios. |
DRAFT: Dense Retrieval Augmented Few-shot Topic classifier Framework (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for few-shot topic classification are limited due to the volume of information pouring in from the Internet . a new framework is proposed to train a classifier for few shot topics . |
| Approach: | They propose a framework to train a classifier for few-shot topic classification using a customized dataset and a dense retriever model. |
| Outcome: | The proposed framework shows superior performance on few-shot topic classification tasks compared to baselines that use in-context learning . |
Enhancing Online Grooming Detection via Backtranslation Augmentation (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models to detect predatory conversations for online conversation platforms are lacking in real-world applications due to sparse distribution of predatory conversation data. |
| Approach: | They propose backtranslation augmentation to augment training datasets with more predatory conversations by using 3 neural translators to augment them. |
| Outcome: | The proposed model improves with fewer training epochs for better classification efficacy on 8 languages from 4 language families and shows that it is more efficient than previous models. |
Typos that Broke the RAG’s Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on the robustness of Large Language Models (LLMs) overlook the interconnected relationships between RAG components or the potential threats prevalent in real-world databases, such as minor textual errors. |
| Approach: | They propose a novel attack method that exploits vulnerabilities in RAG components and tests its robustness against noisy documents. |
| Outcome: | The proposed method devastates the performance of each component and their synergy, and significantly devases the performance. |
GRAFT: A Graph-based Flow-aware Agentic Framework for Document-level Machine Translation (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing Document-level machine translation systems struggle to handle discourse-level phenomena such as pronoun resolution, lexical cohesion, and ellipsis. |
| Approach: | They propose a graph-based document-level machine translation framework that leverages Large Language Models to model translation flow and discourse structure. |
| Outcome: | The proposed framework outperforms commercial and closed systems in eight languages and six domains. |
Overcoming both Domain Shift and Label Shift for Referring Video Segmentation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods to improve the robustness of open-set domain generalization can only recognize seen objects and mark all unseen objects as “unknown” categories . |
| Approach: | They propose a method to make the model maintain good segmentation ability for unknown objects . they propose CLIP-based Reasoning Prompt which can combine text and visual prompts . |
| Outcome: | The proposed method can bridge the gap caused by label shift by combining text and visual prompts to improve text-object matching ability. |
Zero-Shot Conversational Stance Detection: Dataset and Approaches (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing stance detection datasets are limited to a limited set of specific targets . current models are limited in their ability to detect large numbers of unseen targets based on a large number of unidentified targets. |
| Approach: | They propose a speaker interaction and target-aware prototypical contrastive learning model that can detect public opinion towards specific targets using social media data. |
| Outcome: | The proposed model achieves state-of-the-art in zero-shot conversational stance detection with only an F1-macro score of 43.81%. |
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have greatly influenced the development of Large Multi-modal Video Models. |
| Approach: | They propose a benchmark to assess the proficiency of Large Multi-modal Video Models (LMMs) in detecting and localizing anomalies and inconsistencies in videos. |
| Outcome: | The proposed benchmark assesses the proficiency of Video-LMMs in detecting and localizing anomalies and inconsistencies in videos. |
Weakly Supervised Text Classification using Supervision Signals from a Language Model (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing weakly supervised text classification methods require a large number of annotated data and human annotations are expensive. |
| Approach: | They propose to query a masked language model with cloze style prompts to obtain supervision signals. |
| Outcome: | The proposed method outperforms baseline methods on three datasets by 2%, 4%, and 3%. |
Automatic Speech Recognition System-Independent Word Error Rate Estimation (2024.lrec-main)
Copied to clipboard
| Challenge: | Word error rate (WER) is a metric used to evaluate the quality of transcriptions produced by Automatic Speech Recognition systems. |
| Approach: | They propose a hypothesis generation method for ASR system-dependent WER estimation . they use phonetically similar or linguistically more likely alternative words to generate hypotheses . |
| Outcome: | The proposed method outperforms baseline estimators on in-domain data and out-of-domain on Switchboard and CALLHOME. |
Ask To The Point: Open-Domain Entity-Centric Question Generation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | *entity-centric question generation (ECQG) is a task motivated by real-world applications such as topic-specific learning, assisted reading, and fact-checking. |
| Approach: | They propose a PLM-based framework GenCONE with two modules: content focusing and question verification. |
| Outcome: | The proposed framework outperforms baselines and is effective and complementary in generating high-quality questions. |
Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Reasoning models are prone to generating confident, plausible responses that are incorrect (hallucinations). |
| Approach: | They introduce introspective uncertainty quantification to examine whether reasoning models are well-calibrated and does deeper reasoning improve their calibration? |
| Outcome: | The proposed model calibrations show that models are overconfident, overconfent and overconfust with deeper reasoning. |
Operator Selection and Ordering in a Pipeline Approach to Efficiency Optimizations for Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | Natural language processing tasks rely on complex neural models . transformer-based models are typically slow to execute, making it a non-trivial challenge to apply them in real-world applications. |
| Approach: | They propose to consider an efficiency method as an operator applied on a model . they find that the commutativity and cumulativeness of efficiency operators are plausible . |
| Outcome: | The proposed method is commutative and cumulative, and the results are estimated by combining methods. |
Universal Proposition Bank 2.0 (2022.lrec-1)
Copied to clipboard
Ishan Jindal, Alexandre Rademaker, Michał Ulewicz, Ha Linh, Huyen Nguyen, Khoi-Nguyen Tran, Huaiyu Zhu, Yunyao Li
| Challenge: | Semantic role labeling (SRL) is a shallow semantic parsing task that identifies "who did what to whom when, where etc." SRL is useful in a wide range of downstream NLP tasks and real-world applications. |
| Approach: | They propose a method to generate shallow semantic parsing tasks using monolingual SRL and multilingual parallel data. |
| Outcome: | The proposed method improves the quality of the generated propbanks. |
Are You for Real? Detecting Identity Fraud via Dialogue Interactions (D19-1)
Copied to clipboard
| Challenge: | Existing methods to detect identity fraud are prone to errors and are not based on real data. |
| Approach: | They propose to use a KG constructor and structured dialogue management to detect identity fraud in loan applications to generate questions based on personal information. |
| Outcome: | The proposed system can detect fraudsters and achieve higher recognition accuracy compared with rule-based systems. |
CAPSTONE: Composable Attribute‐Prompted Scene Translation for Zero‐Shot Vision–Language Reasoning (2025.emnlp-industry)
Copied to clipboard
| Challenge: | CAPSTONE transforms visual inputs into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Approach: | They propose a plug-and-play framework that transforms off-the-shelf vision models into structured text prompts that can be interpreted by a frozen Large Language Model (LLM). |
| Outcome: | The proposed framework outperforms fully trained VLMs on the POPE dataset while the 4B model achieves competitive results. |
A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)
Copied to clipboard
| Challenge: | Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications. |
| Approach: | They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling . |
| Outcome: | The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point. |
Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning (D19-1)
Copied to clipboard
| Challenge: | Existing studies on video captioning focus on the association relationships between multiple modalities. |
| Approach: | They propose a video captioning model with high-order cross-modal attention (HOCA) they propose low-rank HOCA which adopts tensor decomposition to reduce the space requirement . |
| Outcome: | The proposed model captures cross-modal interaction of different modalities and reduces space requirement. |
Interactive Machine Comprehension with Information Seeking Agents (2020.acl-main)
Copied to clipboard
| Challenge: | Existing machine reading comprehension (MRC) models do not scale effectively to real-world applications like web-level information retrieval and question answering (QA). |
| Approach: | They propose a method that reframes existing machine reading comprehension (MRC) datasets as interactive, partially observable environments. |
| Outcome: | The proposed method "occludes" the majority of a document’s text and adds context-sensitive commands that reveal "glimpses" of the hidden text to a model. |
Adaptive Knowledge-Enhanced Bayesian Meta-Learning for Few-shot Event Detection (2021.findings-acl)
Copied to clipboard
| Challenge: | Event detection typically does not have sufficient labelled data, thus can be formulated as a few-shot learning problem. |
| Approach: | They propose a knowledge-based fewshot event detection method which introduces external event knowledge as the knowledge prior of event types. |
| Outcome: | Experiments show that the proposed method outperforms baselines by 15 F 1 points . event detection is an important task in information extraction . |
Multi-style Generative Reading Comprehension (P19-1)
Copied to clipboard
Kyosuke Nishida, Itsumi Saito, Kosuke Nishida, Kazutoshi Shinoda, Atsushi Otsuka, Hisako Asano, Junji Tomita
| Challenge: | Current studies on generative reading comprehension (RC) focus on extracting an answer span from textual evidence and natural language generation (NLG). |
| Approach: | They propose a multi-style abstractive summarization model for question answering called Masque. |
| Outcome: | The proposed model achieves state-of-the-art performance on the Q&A and Q& A + NLG tasks of MS MARCO and NarrativeQA. |
DeepSolution: Boosting Complex Engineering Solution Design via Tree-based Exploration and Bi-point Thinking (2025.acl-long)
Copied to clipboard
Zhuoqun Li, Haiyang Yu, Xuanang Chen, Hongyu Lin, Yaojie Lu, Fei Huang, Xianpei Han, Yongbin Li, Le Sun
| Challenge: | Existing studies in retrieval-augmented generation (RAG) do not sufficiently address the design of complex engineering solutions. |
| Approach: | They propose a retrieval-augmented generation system that leverages tree-based exploration and bi-point thinking mechanism to generate reliable solutions. |
| Outcome: | Experiments show that the proposed system achieves state-of-the-art (SOTA) performance on the SolutionBench, highlighting its potential to enhance the automation and reliability of complex engineering solution design in real-world applications. |
Evaluating Cultural and Social Awareness of LLM Web Agents (2025.findings-naacl)
Copied to clipboard
Haoyi Qiu, Alexander Fabbri, Divyansh Agarwal, Kung-Hsiang Huang, Sarah Tan, Nanyun Peng, Chien-Sheng Wu
| Challenge: | Existing benchmarks often overlook cultural and social awareness . current evaluations focus on task completion, often ignoring the diverse cultural and socio-cultural backgrounds. |
| Approach: | They propose a benchmark to assess LLM agents’ sensitivity to cultural and social norms across two web-based tasks: online shopping and social discussion forums. |
| Outcome: | The proposed framework evaluates LLM agents’ ability to detect and appropriately respond to norm-violating user queries and observations across two web-based tasks. |
Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements (2025.findings-acl)
Copied to clipboard
Shu Yang, Shenzhe Zhu, Zeyu Wu, Keyu Wang, Junchi Yao, Junchao Wu, Lijie Hu, Mengdi Li, Derek F. Wong, Di Wang
| Challenge: | Existing fraud detection benchmarks focus on single-turn classification tasks, failing to capture dynamic nature of real-world fraud attempts. |
| Approach: | They propose a bilingual benchmark to assess LLMs' ability to resist fraud and phishing attacks across five key fraud categories: Fraudulent Services, Impersonation, Phishing Scams, Fake Job Postings, and Online Relationships. |
| Outcome: | The proposed model improves in role-play settings and in e-commerce and recommendation systems. |
Intuitive or Dependent? Investigating LLMs’ Behavior Style to Conflicting Prompts (2024.acl-long)
Copied to clipboard
| Challenge: | Extensive experiments with seven Large Language Models reveal their varying behaviors. |
| Approach: | They investigate the behaviors of Large Language Models when faced with conflicting prompts versus their internal memory. |
| Outcome: | Extensive experiments with seven LLMs reveal their varying behaviors. |
Text is All You Need: LLM-enhanced Incremental Social Event Detection (2025.acl-long)
Copied to clipboard
| Challenge: | Existing state-of-the-art (SOTA) SED models rely on graph neural networks (GNNs) Existing SED frameworks rely heavily on GNNs, which require complex graph construction and time-consuming training processes. |
| Approach: | They propose a framework that leverages the rich background knowledge of large language models to formalize and disambiguate short texts by completing abbreviations and summarizing informal expressions. |
| Outcome: | The proposed framework outperforms existing models on two challenging real-world datasets. |
R3Mem: Bridging Memory Retention and Retrieval via Reversible Compression (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing memory solutions that store information via parameters struggle with reliable retrieval. |
| Approach: | They propose a memory network that optimizes both information Retention and Retrieval through Reversible context compression. |
| Outcome: | The proposed memory network outperforms conventional memory modules in long-horizon interaction tasks like conversational agents and achieves state-of-the-art performance in language modeling and retrieval-augmented generation tasks. |
TableBank: Table Benchmark for Image-based Table Detection and Recognition (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing techniques for table detection and recognition are limited to document types and layouts. |
| Approach: | They propose to build a table detection and recognition dataset with weak supervision from Word and Latex documents on the internet. |
| Outcome: | The proposed dataset contains 417K high quality labeled tables and is publicly available. |
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that abstractive summarization approaches generate summaries that are not factually consistent with the source document. |
| Approach: | They propose a method that decomposes the document and summary into structured meaning representations (MRs) MRs describe core semantic concepts and their relations, aggregating the main content in both document and summary in a canonical form . |
| Outcome: | The proposed method outperforms existing methods on benchmarks for factuality evaluation. |
Disentangle-based Continual Graph Representation Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing graph embedding methods overlook streaming nature of incoming data in real-world applications. |
| Approach: | They propose a disentangle-based continual graph representation learning framework inspired by the human’s ability to learn procedural knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art continual graph representation learning framework and alleviate catastrophic forgetting problem. |
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs (2026.acl-long)
Copied to clipboard
Xiangyu Zhao, Wanghan Xu, Bo Liu, Yuhao Zhou, Fenghua Ling, Ben Fei, Xiaoyu Yue, Lei Bai, Wenlong Zhang, Xiao-Ming Wu
| Challenge: | Existing datasets often rely on synthetic data or figure-caption pairs, failing to capture the depth and complexity of geoscientific reasoning. |
| Approach: | They propose a multimodal scientific dataset and benchmark curated from open-access publications. |
| Outcome: | MSEarth features over 289K figures with captions enriched by contextual discussions and reasoning from original papers. |
Empirical Study of Zero-shot Keyphrase Extraction with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a prompting-based approach can effectively supersede traditional KE methods, a study shows . our code is available at https://github.com/kangnlp/zero-shot-keyphrase-extraction-with-LLMs. |
| Approach: | They propose four prompting strategies for zero-shot keyphrase extraction using Large Language Models. |
| Outcome: | The proposed prompting strategies outperform state-of-the-art prompting methods on KE benchmark datasets. |
Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agents (2025.emnlp-main)
Copied to clipboard
Haochen Sun, Shuwen Zhang, Lujie Niu, Lei Ren, Hao Xu, Hao Fu, Fangkun Zhao, Caixia Yuan, Xiaojie Wang
| Challenge: | Large Language Models (LLMs) based agent systems have made great strides in real-world applications beyond traditional NLP tasks. |
| Approach: | They propose a new LLM-based Multi-Agent System benchmark, Collab-Overcooked, built on the popular Overcooked-AI game with more applicable and challenging tasks in interactive environments. |
| Outcome: | The proposed benchmark provides a multi-agent framework supporting diverse tasks and objectives and encourages collaboration through natural language communication. |
K-order Ranking Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing list-wise methods focus on optimizing list ranking consistency for LLMs to improve ranking abilities. |
| Approach: | They propose to extend the Plackett-Luce model to accommodate top-K ranking by extending the DPO’s Plact-Lucer model to dynamically determine appropriate K for different samples. |
| Outcome: | The proposed model can be extended to accommodate top-K ranking and improve training efficiency. |
Autoregressive Affective Language Forecasting: A Self-Supervised Task (2020.coling-main)
Copied to clipboard
| Challenge: | Using natural language, we can model emotional language in time based on past changes of language. |
| Approach: | They propose a task of affective language forecasting to model emotional language in time based on past changes of language. |
| Outcome: | The proposed model outperforms existing models on a Twitter dataset of 1,900 users and scores for 6 emotions and 2 additional linguistic attributes. |
Richer Output for Richer Countries: Uncovering Geographical Disparities in Generated Stories and Travel Recommendations (2025.findings-naacl)
Copied to clipboard
| Challenge: | a large body of work examines language models for biases concerning gender, race, occupation and religion . however, the impact of the encoded geographical knowledge on real-world applications has not been documented . |
| Approach: | They examine large language models for two common scenarios that require geographical knowledge: travel recommendations and geo-anchored story generation. |
| Outcome: | The results show that the language models are biased against poorer countries and poorer socioeconomic conditions. |
Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on individual quality and do not assess the value of training data. |
| Approach: | They propose a choice-based sample selection framework that evaluates sample quality . they use LLMs to evaluate the value of each option during the selection process . |
| Outcome: | The proposed model outperforms the full dataset and recent studies on a larger medical dataset. |
How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact (2021.findings-acl)
Copied to clipboard
| Challenge: | Recent years have seen many breakthroughs in natural language processing (NLP), transitioning it from a mostly theoretical field to one with many real-world applications. |
| Approach: | They propose a moral philosophy definition of social good and a framework to evaluate the direct and indirect real-world impact of NLP tasks. |
| Outcome: | The proposed framework evaluates the direct and indirect real-world impact of NLP tasks and adopts the methodology of global priorities research to identify priority causes for NLP research. |
A Survey on Personalized Alignment—The Missing Piece for Large Language Models in Real-World Applications (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their transition to real-world applications reveals a critical limitation: the inability to adapt to individual preferences while maintaining alignment with universal human values. |
| Approach: | They propose a framework that enables LLMs to adapt their behavior within ethical boundaries based on individual preferences. |
| Outcome: | The proposed framework analyzes implementation approaches and evaluates their effectiveness across various scenarios. |
COPR: Continual Human Preference Learning via Optimal Policy Regularization (2025.findings-acl)
Copied to clipboard
Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Zhuo Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) is effective for aligning Large Language Models with human preferences, but its complex process limits its ability to continually learn human feedback. |
| Approach: | They propose a non-RL offline method to convert historical optimal policies into optimization constraints when continually learning new preferences. |
| Outcome: | The proposed method outperforms strong CL baselines in terms of reward-based evaluations and human assessment. |
Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments (2026.acl-long)
Copied to clipboard
Hao Mi, Qiang Sheng, Shaofei Wang, Beizhe Hu, Yifan Sun, Zhengjia Wang, Hengqi Zeng, Yang Li, Danding Wang, Juan Cao
| Challenge: | Existing methods for hallucination detection focus on implicit neural uncertainty or explicit symbolic reasoning, ignoring factual hallucinosities. |
| Approach: | They propose a framework that bridges neural features and symbolic judgments for hallucination detection by leveraging a "meta-judgment" process to map symbolic labels back into the feature space. |
| Outcome: | Extensive experiments on 4 public datasets, across 4 LLMs, against 8 baselines demonstrate the superiority of LaaB. |
UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation (2024.acl-long)
Copied to clipboard
Xun Liang, Shichao Song, Simin Niu, Zhiyu Li, Feiyu Xiong, Bo Tang, Yezhaohui Wang, Dawei He, Cheng Peng, Zhonghao Wang, Haiying Deng
| Challenge: | Large language models (LLMs) produce hallucinated text, compromising their practical utility in professional contexts. |
| Approach: | They have developed an unconstrained hallucination generation evaluation benchmark that contains hallucines generated by large language models with minimal restrictions. |
| Outcome: | The proposed benchmarks are based on a Chinese-language dataset that is lacking in the field. |
Reasoning with Language Model Prompting: A Survey (2023.acl-long)
Copied to clipboard
Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Huajun Chen
| Challenge: | Reasoning is an essential ability for complex problem-solving and can provide back-end support for various real-world applications. |
| Approach: | They present cutting-edge research on reasoning with language model prompting and provide systematic resources to help beginners. |
| Outcome: | The proposed approaches have not been systematically reviewed and analyzed. |
Integrating Data Validation with Large Language Models for Regulation-Guided Tabular Anomaly Detection (2026.acl-long)
Copied to clipboard
| Challenge: | Existing tabular anomaly detection methods focus on detecting anomalies based on data distribution without considering regulatory compliance. |
| Approach: | They propose a task that leverages regulations to detect anomalies in tabular data . they also develop three new datasets to address this task . |
| Outcome: | The proposed method outperforms baselines on three new datasets. |
LUQ: Long-text Uncertainty Quantification for LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing research on Uncertainty Quantification (UQ) predominantly targets short text generation, however, real-world applications often necessitate much longer responses. |
| Approach: | They propose a method that ensembles responses from multiple models and selects the response with the lowest uncertainty. |
| Outcome: | The proposed method outperforms baseline methods in correlating with the model’s factuality scores (negative coefficient of -0.85 observed for Gemini Pro). |
Large Language Models are good multi-lingual learners : When LLMs meet cross-lingual prompts (2025.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that Large Language Models can generate rule-based data in long contexts without following all specified rules. |
| Approach: | They propose a novel prompting strategy Multi-Lingual Prompt which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. |
| Outcome: | The proposed framework outperforms state-of-the-art prompting methods on public datasets across various tasks, with a specific case study in text-to-MIP instances. |
Factual Dialogue Summarization via Learning from Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models generate fluent and coherent summaries, but inconsistencies can be found in generated summary. |
| Approach: | They propose to use symbolic knowledge distillation to improve the factual consistency of smaller pretrained models for dialogue summarization. |
| Outcome: | The proposed model outperforms baseline models in BART, PEGASUS, and Flan-T5 in factual consistency and accuracy. |
Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notes (2024.findings-acl)
Copied to clipboard
Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, Edward Choi
| Challenge: | Clinical notes are an extensive repository of information specific to individual patients. |
| Approach: | They create synthetic large-scale clinical notes using publicly available case reports extracted from biomedical literature and train a clinical large language model, Asclepius. |
| Outcome: | The proposed model outperforms several other models and is supported by detailed evaluations conducted by GPT-4 and medical professionals. |
MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios (2025.findings-acl)
Copied to clipboard
JinYang Huang, Xiachong Feng, Qiguang Chen, Hanjie Zhao, Zihui Cheng, Jiesong Bai, Jingxuan Zhou, Min Li, Libo Qin
| Challenge: | MLDebugging is a benchmark designed to assess debugging challenges within multi-library Python code. |
| Approach: | They propose to introduce a benchmark to assess debugging challenges within multi-library Python code using 126 Python libraries. |
| Outcome: | The proposed benchmark covers 126 Python libraries and a wide range of multi-library code issues. |
Evolutionary Contrastive Distillation for Language Model Alignment (2024.findings-emnlp)
Copied to clipboard
Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi
| Challenge: | Existing studies indicate that large language models struggle with challenging instructions. |
| Approach: | They propose a method for generating high-quality synthetic preference data to enhance the complex instruction-following capability of language models. |
| Outcome: | The proposed method exceeds the performance of current SOTA 7B models and is competitive even with open-source 70B models. |
Benchmarking Data Science Agents (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have emerged as promising data science aids, assisting humans in data analysis and processing. |
| Approach: | They propose an evaluation paradigm and benchmarks that assess the performance of data science agents throughout the entire data science lifecycle. |
| Outcome: | The proposed evaluation paradigm streamlines dataset preparation, improves coverage, and expands benchmarking comprehensiveness. |
CacheNotes: Task-Aware Key-Value Cache Compression for Reasoning-Intensive Knowledge Tasks (2026.eacl-long)
Copied to clipboard
| Challenge: | Current methods for integrating external knowledge into Large Language Models (LLMs) face limitations with broad, multi-source queries, while long-context models are computationally prohibitive. |
| Approach: | They propose a task-aware key-value cache compression method that generates a sequence of CPTs from a corpus and guides a one-time compression of the corpus into a compact, reusable KV cache. |
| Outcome: | The proposed method outperforms Retrieval-Augmented Generation (RAG) on Question-Answering tasks and reduces latency by over 4. |
Open-Vocabulary Federated Learning with Multimodal Prototyping (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing studies assume the label space of training data and test data is identical. |
| Approach: | They propose a framework for adaptation to a federated learning (FL) query that uses arbitrary unknown classes. |
| Outcome: | The proposed framework exploits the knowledge learned from seen classes and robustifies the adapted framework to unseen categories. |
Robust Representation Learning of Biomedical Names (P19-1)
Copied to clipboard
| Challenge: | Biomedical concepts are often mentioned in medical documents under different name variations. |
| Approach: | They propose a framework for learning robust representations of biomedical names and terms . they encode contextual meaning, conceptual meaning, and similarity between synonyms . |
| Outcome: | The proposed framework outperforms baselines on retrieval, similarity and relatedness benchmarks. |
Progra: Progress-Aware Reinforcement Learning for Multi-Turn Function Calling (2026.findings-acl)
Copied to clipboard
Huacan Chai, Zijie Cao, Maolin Ran, Yingxuan Yang, Jianghao Lin, Xin Peng, Hairui Wang, Renjie Ding, Ziyu Wan, Muning Wen, Weiwen Liu, Weinan Zhang, Fei Huang, Ying Wen
| Challenge: | Existing methods for multi-turn function calling are limited by redundancy and lack explicit integration of progress awareness into training. |
| Approach: | They propose a framework that explicitly integrates progress awareness into LLM training for multi-turn function calling. |
| Outcome: | Empirical results show that Progra outperforms existing methods on two public benchmarks. |
Conceptual Pacts for Reference Resolution Using Small, Dynamically Constructed Language Models: A Study in Puzzle Building Dialogues (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing large language models can be fine-tuned offline but are large and resource-intensive. |
| Approach: | They propose to use a simple reference resolver to simulate a conceptual pact process over time with different conversation pairs. |
| Outcome: | The proposed model performs better than a pre-trained model with exhaustive retraining after each prediction, while being more transparent, faster and less resource-intensive. |
GVdoc - Graph-based Visual DOcument Classification (2023.findings-acl)
Copied to clipboard
| Challenge: | Visual document classifiers have shown impressive performance on in-distribution test sets, but they have a hard time classifying out-of-difference examples. |
| Approach: | They propose a graph-based document classification model that generates a document graph based on its layout and trains a neural network to learn node and graph embeddings. |
| Outcome: | The proposed model outperforms state-of-the-art models on out-of distribution data while maintaining comparable performance on in-distribution data. |
Continual Few-shot Event Detection via Hierarchical Augmentation Networks (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for continual few-shot event detection use labeled data, but in real-world applications, new event types emerge continually. |
| Approach: | They propose a memory-based framework for continual few-shot event detection . they incorporate prototypical augmentation into the memory set to memorize previous event types . |
| Outcome: | The proposed method outperforms existing methods in multiple continual few-shot event detection tasks. |
Red-Teaming LLM Multi-Agent Systems via Communication Attacks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Model-based Multi-Agent Systems (LLM-MAS) have revolutionized complex problem-solving capability by enabling agent collaboration through message-based communications. |
| Approach: | They propose an attack that exploits communication mechanisms in Large Language Model-based Multi-Agent Systems (LLM-MAS) by intercepting and manipulating inter-agent messages. |
| Outcome: | The proposed attack exploits communication mechanisms in large language model-based multi-agent systems by intercepting and manipulating inter-agencies. |
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue systems cannot guarantee that all user needs are taken into account in the design phase. |
| Approach: | They propose a new incremental learning framework to design task-oriented dialogue systems without pre-defining user needs. |
| Outcome: | The proposed framework is robust to unconsidered user actions and can update itself online with less annotation cost. |
ReLM: Leveraging Language Models for Enhanced Chemical Reaction Prediction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for predicting chemical reactions are limited by insufficient training data and inability to utilize textual information. |
| Approach: | They propose a framework that leverages chemical knowledge encoded in language models to assist GNNs, thereby enhancing the accuracy of real-world chemical reaction predictions. |
| Outcome: | The proposed framework improves state-of-the-art GNN-based methods across chemical reaction datasets especially in out-of distribution settings. |
Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Current sycophancy research has largely overlooked its specific manifestations in the video-language domain. |
| Approach: | They propose a video-LLM sycophancy benchmarking and evaluation to evaluate scophancies in video-LLMs. |
| Outcome: | The proposed benchmark evaluates sycophantic behavior in state-of-the-art Video-LLMs across diverse question formats, prompt biases, and visual reasoning tasks. |
Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic Models (2025.acl-long)
Copied to clipboard
Zongxia Li, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Paiheng Xu, Daniel Kofi Stephens, Juan Francisco Fung, Alden Dima, Jordan Lee Boyd-Graber
| Challenge: | a common use of NLP is to facilitate the understanding of large document collections. |
| Approach: | They propose to use large language models to replace probabilistic topic models in real-world applications. |
| Outcome: | The proposed model generates more human-readable topics and shows higher average win probabilities than traditional models for data exploration. |
Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies show that neural natural language processing models are vulnerable to backdoor attacks. |
| Approach: | They propose to inject neural models with backdoors activated by word substitution . their results raise a serious alarm to the security of NLP models, they argue . |
| Outcome: | The proposed backdoors are activated by a learnable combination of word substitution and exhibit higher invisibility than previous methods. |
GAMEBoT: Transparent Assessment of LLM Reasoning in Games (2025.acl-long)
Copied to clipboard
| Challenge: | Existing efforts to create benchmarks that move beyond superficial pattern recognition to delve into the profound reasoning skills required for problemsolving face challenges such as insufficient interpretability, performance saturation or data contamination. |
| Approach: | They propose a gaming arena designed for rigorous assessment of LLM reasoning capabilities. |
| Outcome: | The proposed framework decomposes complex reasoning into predefined modular subproblems and generates ground truth for these subproblem types. |
Text Segmentation by Cross Segment Attention (2020.emnlp-main)
Copied to clipboard
| Challenge: | Document and discourse segmentation are two fundamental NLP tasks pertaining to breaking up text into constituents. |
| Approach: | They propose three transformer-based NLP models that break up text into constituents and compare them to previous approaches. |
| Outcome: | The proposed architectures reduce errors by a large margin on three datasets and improve performance on real-world datasets. |
FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answering (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on single-hop, single-modality, or short texts, limiting real-world applications . despite advances in visual question answering, this multihop setting remains underexplored due to a lack of quality datasets. |
| Approach: | They propose a framework for creating a high-quality dataset for multimodal multihop question answering . they use a 5-stage pipeline to acquire relevant multimodal documents from Wikipedia . |
| Outcome: | The proposed framework outperforms existing methods on multimodal multihop question answering datasets. |
Tchebycheff Procedure for Multi-task Text Classification (2020.acl-main)
Copied to clipboard
| Challenge: | Existing methods for text classification assume that multitask text classification problems are convex multiobjective optimization problems. |
| Approach: | They propose a Tchebycheff procedure to optimize multi-task classification problems without convex assumption. |
| Outcome: | The proposed method is able to find an arbitrary Pareto optimal solution in the PareTO set if the problem is convex, but excludes many Paret optimal solutions from its search scope. |
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement (2024.findings-acl)
Copied to clipboard
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipalli, Michael Mahoney, Kurt Keutzer, Amir Gholami
| Challenge: | Pretrained large language models are currently state-of-the-art for solving most tasks . however, many of them are in the low-data regime, making fine-tuning challenging . a new data augmentation strategy uses a teacher LLM to augment a small seed dataset . |
| Approach: | They propose a targeted and iterative data augmentation strategy that augments a teacher LLM to fine-tune a small seed dataset by adding additional data. |
| Outcome: | The proposed approach outperforms fine-tuning and other data augmentation strategies on a small seed dataset. |
From Moments to Milestones: Incremental Timeline Summarization Leveraging Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Prior work on timeline summarization has neglected the potential synergy between the two forms of timelines. |
| Approach: | They propose a timeline summarization approach that leverages large language models to generate both event and topic timelines. |
| Outcome: | The proposed approach outperforms the best existing approaches in four TLS benchmarks. |
Does Gender Matter? Towards Fairness in Dialogue Systems (2020.coling-main)
Copied to clipboard
| Challenge: | Recent studies have shown that AI is unfair in many real-world applications such as computer vision and recommendations. |
| Approach: | They propose to use a benchmark dataset to study the fairness of dialogue systems to understand their bias. |
| Outcome: | The proposed methods reduce the bias in dialogue systems significantly. |
Does Chain-of-Thought Reasoning Help Mobile GUI Agents? An Empirical Study (2026.findings-acl)
Copied to clipboard
| Challenge: | Reasoning capabilities have improved vision-language models in domains like math, coding, and visual question-answering, but their impact on real-world applications remains unclear. |
| Approach: | They evaluate six pairs of VLMs by comparing their base and reasoning-enhanced versions across static and interactive benchmarks. |
| Outcome: | The reasoning-enhanced models perform better on static and interactive benchmarks than non-reasoning models. |
MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues (2026.findings-acl)
Copied to clipboard
Yaning Pan, Qianqian Xie, Guohui Zhang, Zekun Moore Wang, Yongqian Wen, Yuanxing Zhang, Haoxuan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, Yonghong Lin, An Ping, Shihao Li, Yanghai Wang, Tianhao Peng, Jiaheng Liu
| Challenge: | Existing evaluation benchmarks for Multimodal Large Language Models (MLLMs) focus on single-turn question answering, overlooking the complexity of multi-turn dialogues in real-world scenarios. |
| Approach: | They propose a video understanding benchmark for MLLMs in multi-turn dialogues that assesses six core competencies that focus on perceptivity and interactivity. |
| Outcome: | The MT-Video-Bench evaluates 1,000 multi-turn dialogues from diverse domains and reveals significant performance discrepancies and limitations in handling multi-turned video dialogues. |
VISREAS: Complex Visual Reasoning with Unanswerable Questions (2024.findings-acl)
Copied to clipboard
| Challenge: | Logic2Vision is a visual question-answering dataset that validates question authenticity with the corresponding image and then reasoning over it. |
| Approach: | They propose a compositional visual question-answering dataset, VisReas, that consists of answerable and unanswerable visual queries . they use visual genome scene graphs to generate the query and the reasoning steps to generate it. |
| Outcome: | The proposed model outperforms generative models and the existing classification models and outperformed existing models. |
Understanding the Inner-workings of Language Models Through Representation Dissimilarity (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dissimilarity measures measure the extent to which two model’s internal representations differ . they can identify and locate generalization properties of models that are invisible via in-distribution test set performance. |
| Approach: | They propose to use representation dissimilarity measures to measure the extent to which two model’s internal representations differ. |
| Outcome: | The proposed dissimilarity measures can identify and locate generalization properties of models that are invisible via in-distribution test set performance and new evaluations of how language model features vary as width and depth are increased. |
Nearest Neighbor Knowledge Distillation for Neural Machine Translation (2022.naacl-main)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a state-of-the-art machine translation technique . however, it requires conducting kNN searches for each decoding step, which increases the cost of decoding . |
| Approach: | They propose to move the time-consuming kNN search forward to the preprocessing phase and introduce k Nearest Neighbor Knowledge Distillation (kNN-KD) that trains the base NMT model to directly learn the knowledge of kN. |
| Outcome: | The proposed method improves over the state-of-the-art model while maintaining the same training and decoding speed as the standard model. |
RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework (2025.acl-long)
Copied to clipboard
Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing evaluation metrics for RAG systems are lacking due to high costs of data construction and lack of factual accuracy. |
| Approach: | They propose a framework to evaluate RAG systems in specialized scenarios . they propose three new metrics to evaluate LLM-generated responses . |
| Outcome: | The proposed framework outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples. |
Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling (2022.acl-long)
Copied to clipboard
| Challenge: | Prior work on text generation models focused on new architectures for permuted document tasks. |
| Approach: | They propose to use a basic model architecture to improve coherence evaluation of machine generated text. |
| Outcome: | The proposed model improves on a task-independent test set and shows significant improvements in coherence evaluations of downstream tasks. |
Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges (2025.acl-long)
Copied to clipboard
Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank
| Challenge: | linguistics studies how context influences meaning of language and how people use it to convey implied meanings, emotions, and intentions. |
| Approach: | They analyze task designs, data collection methods, evaluation approaches and their relevance to real-world applications. |
| Outcome: | The findings highlight emerging trends, challenges, and gaps in existing benchmarks . the findings will contribute to more nuanced and context-aware NLP models . |
DepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for spatial reasoning in text overlook the gap between natural language and symbolic structures. |
| Approach: | They propose a novel depth-wise Graph Neural Network to aggregate spatial information over the depth dimension instead of the breadth dimension of the graph. |
| Outcome: | The proposed model outperforms existing methods on two multi-hop spatial reasoning datasets. |
On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation (2025.findings-naacl)
Copied to clipboard
| Challenge: | Hallucination is a popular topic in natural language generation (NLG). |
| Approach: | They propose to use large language models to evaluate faithfulness of guided NLGs by a rubric template and large language inference models to score the generation on quantifiable scales. |
| Outcome: | The proposed system can provide accurate judgement and explain whether a source and generation are factually consistent. |
CSS: A Large-scale Cross-schema Chinese Text-to-SQL Medical Dataset (2023.findings-acl)
Copied to clipboard
| Challenge: | a cross-domain text-to-SQL task aims to parse user questions into SQL on complete unseen databases . a single-domain task evaluates the performance on identical databases based on the same domain . |
| Approach: | They propose a cross-domain text-to-SQL task that parses user questions into SQL on unseen databases. |
| Outcome: | The proposed system can parse user questions into SQL on complete unseen databases. |
REALM: A Dataset of Real-World LLM Use Cases (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on LLM adoption and their social implications lack empirical grounding, weakening their validity. |
| Approach: | They propose to integrate a dataset of over 94,000 LLM use cases collected from Reddit and news articles to provide insights into LLM adoption across different domains. |
| Outcome: | The proposed dataset includes over 94,000 LLM use cases collected from Reddit and news articles. |
Intra-/Inter-Interaction Network with Latent Interaction Modeling for Multi-turn Response Selection (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for multi-turn response selection are not practical as the turns of conversations vary. |
| Approach: | They propose to use latent interaction modeling to model multi-level interactions between utterance and response. |
| Outcome: | The proposed method outperforms state-of-the-art methods on three multi-turn response selection benchmark datasets. |
Diffusion Based Counterfactual Augmentation for Dual Sentiment Classification (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods to augment sentiment models have failed to mitigate spurious association problem inherent in the original data. |
| Approach: | They propose a framework for enhancing sentiment models using an antonymous paradigm and contrastive learning to generate high-quality samples. |
| Outcome: | The proposed framework achieves state-of-the-art performance on four benchmark datasets. |
A Practical Analysis of Human Alignment with *PO (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior research focused on identifying the best-performing method to varying hyperparameters . prior research focused primarily on a grid search, which can be impractical for general practitioners . |
| Approach: | They propose a preference optimization method that is more stable across hyperparameters and reduces the average response length. |
| Outcome: | The proposed method increases likelihood of achieving better results through various metrics, such as KL divergence and response length. |
From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained language models have shown impressive performance in various language tasks, but are prone to spurious correlations and illusory information. |
| Approach: | They propose to use pre-trained language models to justify decisions with formalized, coherent reasoning chains. |
| Outcome: | The proposed strategies improve coherence of rationalizations yielding state-of-the-art results on Tiered Reasoning for Intuitive Physics (TRIP). |
READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises (2023.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for Chinese inputs often lack a realistic representation of real-world noises. |
| Approach: | They construct a Chinese multi-task benchmark with REalistic and Diverse input noises . they use pinyin input and speech input to recruit speakers from diverse dialects based on their inputs - a feature that is important for Chinese NLP benchmarks if it is implemented in real-world applications. |
| Outcome: | The proposed benchmarks are based on four different tasks and are designed to maximize diversity. |
R2D2: Robust Data-to-Text with Replacement Detection (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to mitigate unfaithful text generation are inadequate . data-to-text generation requires a structured input format . |
| Approach: | They propose a training framework that addresses unfaithful Data-to-Text generation by training a system as a generator and faithfulness discriminator with additional replacement detection and unlikelihood learning tasks. |
| Outcome: | The proposed training framework improves FeTaQA, LogicNLG, and ToTTo fidelity on D2T systems. |
Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards (2024.acl-long)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) relies on scalar rewards to capture user preferences. |
| Approach: | They propose a framework that integrates multi-objective reward modeling to represent diverse preference profiles. |
| Outcome: | The proposed method improves performance across reward objectives and targets. |
Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration (2026.acl-long)
Copied to clipboard
| Challenge: | Existing context window extension methods obstruct scaling external knowledge input. |
| Approach: | They develop a multi-agent framework to overcome two core bottlenecks in existing agent orchestration designs. |
| Outcome: | The proposed framework overcomes two core bottlenecks and improves inference-time knowledge integration without longer-context training. |
HEAL: A Hypothesis-Based Preference-Aware Analysis Framework (2025.findings-emnlp)
Copied to clipboard
Yifu Huo, Chenglong Wang, Qiren Zhu, Shunjie Xing, Tong Xiao, Chunliang Zhang, Tongran Liu, JingBo Zhu
| Challenge: | Preference optimization methods like DPO are often evaluated on a single response, overlooking other outputs. |
| Approach: | They propose a Hypothesis-based PrEference-aware AnaLysis Framework that formulates preference alignment as a re-ranking process within hypothesis spaces. |
| Outcome: | The proposed evaluation paradigm re-ranks preference alignment as a reranking process within hypothesis spaces. |
Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications (2021.emnlp-main)
Copied to clipboard
| Challenge: | Sentence-level Quality estimation (QE) is traditionally a regression task . but large multilingual contextualized language models are expensive and infeasible for real-world applications. |
| Approach: | They evaluate several model compression techniques for QE and find they are inefficient . they argue that a full model parameterization is required to achieve SoTA results . |
| Outcome: | The proposed models are poorly expressive in a regression task, the authors argue . they show that reframing QE as a classification problem and evaluating models would improve their performance in real-world applications. |
Large Language Models are Students at Various Levels: Zero-shot Question Difficulty Estimation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advancements in educational platforms have emphasized the importance of personalized education. |
| Approach: | They propose a framework that utilizes large language models to represent students at various levels to estimate question difficulty with and without student question-solving records. |
| Outcome: | The proposed framework outperforms baseline models on the DBE-KT22 and ASSISTMents 2005–2006 benchmarks and shows a high correlation with the regressed IRT curve. |
Pre-trained Language Models for Entity Blocking: A Reproducibility Study (2024.naacl-long)
Copied to clipboard
| Challenge: | Entity Resolution (ER) is an essential task in data integration . state-of-the-art neural IR models that are based on large language models have not been evaluated on the ER task. |
| Approach: | They evaluate state-of-the-art neural IR models that are based on large language models on a wide range of real-world datasets and evaluate their generalization abilities. |
| Outcome: | The proposed methods have been evaluated on a wide range of datasets and their generalization abilities. |
StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation benchmarks focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. |
| Approach: | They propose a multi-turn instruction following benchmark with structural flow modeling that defines an innovative structural flow framework with six fundamental inter-turn relationships. |
| Outcome: | The proposed model is based on a framework with six fundamental inter-turn relationships and is able to analyze and generate specific dialogue flows tailored to specific scenarios. |
Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation methods for large language models overlook the inherent randomness of LLMs. |
| Approach: | They propose a hierarchical statistical model that incorporates both benchmark characteristics and LLM randomness to provide a more comprehensive representation of benchmarking process. |
| Outcome: | The proposed model improves the accuracy of estimating the benchmark score and reduces variance. |
Learning to Discriminate Perturbations for Blocking Adversarial Attacks in Text Classification (D19-1)
Copied to clipboard
| Challenge: | Existing studies on adversarial attacks on deep learning models focus on generation of adversarials and defense against adversarial attacks. |
| Approach: | They propose a framework to identify and adjust malicious perturbations and block adversarial attacks for machine learning models. |
| Outcome: | The proposed framework outperforms baseline methods in blocking adversarial attacks for text classification models. |
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a large pre-trained language model can cause computational burdens in inference time due to multiple forward passes. |
| Approach: | They propose a method to learn fixed text representations with source tasks . they learn a task-specific prefix for each source task independently and combine them . |
| Outcome: | The proposed method improves generalizability of representations with source tasks. |
Enhance Robustness of Language Models against Variation Attack through Graph Integration (2024.lrec-main)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are used in many NLP applications but their vulnerability to adversarial attacks can lead to false or misleading information being distributed. |
| Approach: | They propose a method to incorporate a Chinese character variation graph into pre-trained language models to increase their robustness against character variation attacks in Chinese content. |
| Outcome: | The proposed method outperforms existing language models in combating adversarial attacks in Chinese content. |
When Long Helps Short: How Context Length in Supervised Fine-tuning Affects Behavior of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have achieved impressive performance across NLP tasks. |
| Approach: | They propose to use long-context SFT to improve short-contemporary performance . they also decouple and analyze two key components, Multi-Head Attention and Feed-Forward Network . |
| Outcome: | The proposed model improves short-context performance, contrary to pretraining. |
Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for extending the maximum context lengths of language models are lacking a strong baseline for in-context few-shot classification and on more challenging Chain-of-Thought reasoning, such as HotpotQA, deteriorate question miscomprehension and false inference. |
| Approach: | They propose to harness window-wise attention and positional embedding techniques to extend the maximum context lengths of language models. |
| Outcome: | The proposed method is able to extend the maximum context lengths of language models, e.g., 2048 for LLaMA, by harnessing window-wise attention and positional embedding techniques. |
Revisiting Transformer-based Models for Long Document Classification (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent literature in text classification is biased towards short text sequences . multi-page multi-paragraph documents cannot be efficiently encoded by vanilla transformers based on short text. |
| Approach: | They compare different Transformer-based Long Document Classification approaches to mitigate the computational overhead of vanilla transformers to encode much longer text. |
| Outcome: | The proposed models can process longer text and provide practical advice for long document classification tasks. |
Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Large transformer models are expensive and slow to use in many applications. |
| Approach: | They propose an efficient neural network to distill large transformers into a single smaller model. |
| Outcome: | The proposed model outperforms existing models on English datasets . it outperformed existing models with 2.7 more parameters and 2.5 slower . |
HiSMatch: Historical Structure Matching based Temporal Knowledge Graph Reasoning (2022.findings-emnlp)
Copied to clipboard
Zixuan Li, Zhongni Hou, Saiping Guan, Xiaolong Jin, Weihua Peng, Long Bai, Yajuan Lyu, Wei Li, Jiafeng Guo, Xueqi Cheng
| Challenge: | Temporal Knowledge Graphs (TKGs) store facts as triples in the form of subject, relation, object, timestamps. |
| Approach: | They propose a Temporal Knowledge Graph (TKG) model that extends each triple with a timestamp to describe dynamic facts. |
| Outcome: | The proposed model improves on six benchmark datasets with up to 5.6% performance improvement compared to the state-of-the-art models. |
SymKGQA: Few-Shot Knowledge Graph Question Answering via Symbolic Program Generation and Execution (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models have led to low-level LFs that are limited to the knowledge of underlying LLM about the LF. |
| Approach: | They propose a framework that generates a symbolic LF in a few-shot setting using Large Language Models. |
| Outcome: | The proposed framework outperforms all other few-shot and many fully-supervised KGQA approaches. |
Disambiguating Reference in Visually Grounded Dialogues through Joint Modeling of Textual and Multimodal Semantic Structures (2025.acl-long)
Copied to clipboard
| Challenge: | incorporating textual and multimodal reference resolution improves performance in visual-based reference resolution . Phrase grounding is a well-established task for understanding semantic relations between mentions and objects . ambiguities caused by pronouns and ellipses can arise in visually grounded dialogues . |
| Approach: | They propose a framework that unifies textual and multimodal reference resolution by mapping mention embeddings to object embeddements and selecting mentions or objects based on their similarity. |
| Outcome: | The proposed framework performs better in phrase grounding than other models for this task. |
P²Net: Parallel Pointer-based Network for Key Information Extraction with Complex Layouts (2025.findings-acl)
Copied to clipboard
Kaiwen Wei, Jie Yao, Jiang Zhong, Yangyang Kang, Jingyuan Zhang, Changlong Sun, Xin Zhang, Fengmao Lv, Li Jin
| Challenge: | Existing methods for key information extraction are based on a limited set of entity categories and fixed layouts. |
| Approach: | They propose a large-scale, human-annotated dataset for key information extraction . it is based on a human-annotated layout and 1,162 entity categories . they propose 'parallel pointer-based network' that leverages implicit relationships . |
| Outcome: | Experiments on widely-used datasets show that the proposed model outperforms state-of-the-art methods while maintaining fast inference speeds. |
GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely deployed as autonomous agents capable of following user instructions and making decisions in real-world applications. |
| Approach: | They propose a benchmark to evaluate LLMs' ability to follow domain-oriented guidelines . they evaluate Lms on three critical aspects: adherence to diverse rules, robustness to rule updates . |
| Outcome: | The proposed benchmark evaluates LLMs on three critical aspects: adherence to diverse rules, robustness to rule updates, and alignment with human preferences. |
CAPE: Context-Aware Personality Evaluation Framework for Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies use a context-free approach to assess humans . existing studies use the Disney World test, which ignores real-world applications . |
| Approach: | They propose a framework to assess personality traits in large language models . they use conversational history to quantify the consistency of LLM responses . |
| Outcome: | The proposed framework improves consistency of responses in large language models . it also shows that conversational history enhances consistency and personality shifts . |
Language Detoxification with Attribute-Discriminative Latent Space (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to detoxify toxic text require excessive memory, computations and time. |
| Approach: | They propose a method to generate toxic text using an attribute-discriminative latent space. |
| Outcome: | The proposed method outperforms baselines on detoxified language and dialogue generation tasks while being time- and memory-efficient. |
SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation (2025.acl-long)
Copied to clipboard
Wenyu Zhang, Wei En Ng, Lixin Ma, Yuwen Wang, Junqi Zhao, Allison Koenecke, Boyang Li, Wanglu Wanglu
| Challenge: | Current vision-language models lack multi-dimensional spatial reasoning capabilities for human-like understanding and applications. |
| Approach: | They propose a hierarchical evaluation framework that probes models across increasing levels of complexity and integrates spatial, visual, and logical understanding. |
| Outcome: | The proposed framework probes models across increasing levels of complexity, from basic skills to multi-skill integration and high-level reasoning that combines spatial, visual, and logical understanding. |
ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline (2024.findings-emnlp)
Copied to clipboard
Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Zhao Wenyi, Jie Tang, Yuxiao Dong
| Challenge: | Large language models (LLMs) have shown excellent mastering of human language but struggle in real-world applications that require mathematical problem-solving. |
| Approach: | They propose a pipeline to train a general Math-Critique model from the LLM itself to provide feedback signals and employ rejective fine-tuning and direct preference optimization over the Llm's own generations for data collection. |
| Outcome: | The proposed pipeline outperforms existing LLMs that could be two times larger. |
UMRSpell: Unifying the Detection and Correction Parts of Pre-trained Models towards Chinese Missing, Redundant, and Spelling Correction (2023.acl-long)
Copied to clipboard
| Challenge: | Chinese Spelling Correction (CSC) is a task of detecting and correcting misspelled charac- ters in Chinese texts. |
| Approach: | They propose a model to learn detection and correction parts together from a multi-task learning perspective. |
| Outcome: | The proposed model can learn detection and correction parts together from a multi-task learning perspective. |
KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus With More Data, Speakers, and Topics (2022.lrec-1)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) is a process of converting written text into speech. |
| Approach: | They present an expanded version of their text-to-speech corpus for Kazakh . they propose to use the corpus to build high-quality TTS systems for the language . |
| Outcome: | The constructed corpus is sufficient to build robust TTS models for Kazakh and other Turkic languages, with a subjective mean opinion score ranging from 3.6 to 4.2 for all the five speakers. |
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on temporal sentence grounding rely on expensive video-query paired annotations . despite this, there are no ground-truth annotations in the current work . |
| Approach: | They propose to use paired video-query and segment boundary annotations to generate temporal sentence grounding without training. |
| Outcome: | The proposed model outperforms existing unsupervised methods and beats supervised ones on two challenging datasets. |
Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content Detectors (2024.acl-long)
Copied to clipboard
| Challenge: | Recent research indicates that AI-text detection systems lack robustness and struggle to effectively differentiate perturbed texts. |
| Approach: | They propose to evaluate the robustness of current detection systems by using black-box text perturbation methods and adversarial learning experiments. |
| Outcome: | The proposed methods assess the robustness of current detection models across perturbation granularities and the impact of perturbation data augmentation on the robustity of AI-text detectors. |
Cognitive-Level Adaptive Generation via Capability-Aware Retrieval and Style Adaptation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models struggle to adapt content to users with differing cognitive capacities, leading to cognitive misalignment. |
| Approach: | They propose a cognitive-level alignment framework that aligns both knowledge complexity and presentation style with user cognition. |
| Outcome: | The proposed framework aligns knowledge complexity and presentation style with user cognition. |
Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey (2024.acl-long)
Copied to clipboard
Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, 林一冠 林一冠, Bin Xu, Ren Bowen, Chong Feng, Yang Gao, Heyan Huang
| Challenge: | Existing benchmark-based evaluations cannot accurately reflect the performance of real-world applications. |
| Approach: | They propose a reliable strategy for domains to choose more robust LLMs for real-world applications. |
| Outcome: | The proposed strategy addresses the challenges faced by domains to choose more robust LLMs for real-world applications. |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |
IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Recent approaches to quantify uncertainty in LLMs produce short or constrained answer sets, but many real-world applications require long-form and free-form text generation. |
| Approach: | They propose a framework that leverages inter-sample consistency and intra-sampled faithfulness to quantify the uncertainty in long-form LLM outputs. |
| Outcome: | The proposed framework provides reliable measures of claim-level uncertainty and the model’s faithfulness over two widely used long-form generation datasets. |
A Novel Table-to-Graph Generation Approach for Document-Level Joint Entity and Relation Extraction (2023.acl-long)
Copied to clipboard
| Challenge: | Existing document-level relation extraction methods assume entities and their mentions are given beforehand, which is inadequate for real-world applications. |
| Approach: | They propose a table-to-graph generation model for joint extraction of entities and relations at document-level. |
| Outcome: | The proposed model surpasses existing methods by a large margin and achieves state-of-the-art results on a document-level relation extraction dataset. |
A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing annotated training datasets hinder development of supervised learning models for multimedia content . lack of annotating benchmarks hinders development of models with satisfactory performance . a recent study shows that large language models have zero-shot performance in multimedia understanding . |
| Approach: | They propose to verbalize long videos to generate their descriptions in natural language . they then perform video-understanding tasks on the generated story as opposed to the original video . |
| Outcome: | The proposed method achieves better results than baselines for video understanding. |
Feature Structure Matching for Multi-source Sentiment Analysis with Efficient Adaptive Tuning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing domain matching methods tend to pull all feature instances close, but they are expensive and expensive to update. |
| Approach: | They propose to extract multi-layer features from a large pre-trained model and propose a dynamic parameter fusion module to exploit them for efficient and adaptive tuning. |
| Outcome: | The proposed framework is more robust and generalizable in the multi-source scenario. |
Exploring Graph Representations of Logical Forms for Language Modeling (2025.findings-acl)
Copied to clipboard
| Challenge: | Graph-based formal-logical distributional semantics models are more data-efficient than textual counterparts. |
| Approach: | They propose a pretrained language model over graph representations of logical forms as a proof-of-concept. |
| Outcome: | The proposed model outperforms textual, transformer LMs on downstream tasks . the model is likely to scale with additional parameters and pretraining data . |
ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models (2024.emnlp-main)
Copied to clipboard
Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, Hayato Yamana
| Challenge: | Currently, tool-augmented large language models (LLMs) only achieve total scores of 45.3 and 37.0, respectively, on a scale of 100. |
| Approach: | They propose a multi-level diagnostic process to assess the LLM's hallucinations through two perspectives: depth and breadth. |
| Outcome: | The proposed diagnostic process assesses the hallucinations of large language models through two perspectives: depth and breadth. |
Concept2Box: Joint Geometric Embeddings for Learning Two-View Knowledge Graphs (2023.findings-acl)
Copied to clipboard
Zijie Huang, Daheng Wang, Binxuan Huang, Chenwei Zhang, Jingbo Shang, Yan Liang, Zhengyang Wang, Xian Li, Christos Faloutsos, Yizhou Sun, Wei Wang
| Challenge: | Existing methods to embed knowledge graphs have ignored the fact that they contain two fundamentally different views: high-level ontology-view concepts and fine-grained instance-view entities. |
| Approach: | They propose a novel geometric representation that jointly embeds the two views of a KG using dual geometric representations. |
| Outcome: | Experiments on the public DBpedia KG and a newly-created industrial KG show the proposed method works well. |
From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models and datasets for training summarization models are limited for less resourceful languages like Hungarian . |
| Approach: | They propose to use a Hungarian corpus for training abstractive and extractive summarization models by cleaning, preprocessing and deduplication. |
| Outcome: | The proposed model trains abstractive and extractive summarization models using the dataset . it will be made publicly available, encouraging replication, further research, and real-world applications across various domains. |
Exploiting Instruction-Following Retrievers for Malicious Information Retrieval (2025.findings-acl)
Copied to clipboard
| Challenge: | Instruction-following retrievers are increasingly used in real-world applications, but little research has investigated the safety risks associated with their increasing search capabilities. |
| Approach: | They investigate the ability of retrievers to satisfy malicious queries . they find that for >50% of queries, retrievers can select harmful passages . |
| Outcome: | The findings highlight the safety risks associated with instruction-following retrievers . they show that even safety-aligned LLMs can satisfy malicious requests . |
Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing pruning algorithms suffer from limitations such as architecture specificity and reliance on demanding calculations. |
| Approach: | They propose a pruning algorithm based on Kernel Density Estimation . it preserves most significant parameters while restoring others to their pre-training state . |
| Outcome: | The proposed pruning algorithm achieves better performance than the original unpruned version. |
How does Misinformation Affect Large Language Model Behaviors and Preferences? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies have explored the role of Large Language Models in combating misinformation, but there is still a lack of detailed analysis on the specific aspects and extent to which LLMs are influenced by misinformation. |
| Approach: | They propose to use a benchmark to evaluate LLMs' behavior and knowledge preference toward misinformation to identify their models. |
| Outcome: | The proposed approach is based on 10,346,712 pieces of misinformation and examines knowledge conflicts and stylistic variations. |
TL-CL: Task And Language Incremental Continual Learning (2024.emnlp-main)
Copied to clipboard
| Challenge: | a multilingual model is periodically updated to accommodate new tasks in previously learned languages or new languages for established tasks. |
| Approach: | They propose an adapter-based parameter-efficient fine-tuning strategy for continual learning in multilingual models. |
| Outcome: | The proposed approach outperforms other parameter-efficient approaches without access to historical data for replay. |
Spatial-Agent: Agentic Geo-spatial Reasoning with Scientific Core Concepts (2026.acl-long)
Copied to clipboard
| Challenge: | Existing LLM-based agents lack inherent spatial awareness, relying on web search or text matching while hallucinating spatial relationships. |
| Approach: | They propose a spatial-based agent that can perform real-world geospatial computations . they use natural-language questions to parse into executable workflows based on geoFlow Graphs - directed acyclic graphs with nodes corresponding to spatial concepts and edges representing transformations. |
| Outcome: | The proposed agent outperforms existing baselines on MapEval-API and MapQA benchmarks while producing interpretable and executable geospatial workflows. |
Multi-task Adversarial Attacks against Black-box Model with Few-shot Queries (2025.acl-long)
Copied to clipboard
| Challenge: | Existing adversarial text attacks rely on abundant access to shared internal features and numerous queries, limited to a single task type. |
| Approach: | They propose a black-box attack that exploits the transferability of adversarial texts . they use a deep-level substitute model trained in a plug-and-play manner for text classification . |
| Outcome: | The proposed attack can target multiple tasks with minimal perturbations . it can target commercial APIs, large language models, and image-generation models . |
FaithLens: Detecting and Explaining Faithfulness Hallucination (2026.findings-acl)
Copied to clipboard
Shuzheng Si, Qingyi Wang, Haozhe Zhao, Yuzhuo Bai, Guanqiao Chen, Kangyang Luo, Gang Chen, Fanchao Qi, Minjia Zhang, Baobao Chang, Maosong Sun
| Challenge: | Recent progress in large language models (LLMs) has revolutionized text generation. |
| Approach: | They propose a faithfulness hallucination detection model that can provide binary predictions and corresponding explanations to improve trustworthiness. |
| Outcome: | The proposed model outperforms advanced models on 12 diverse tasks. |
Structured Pruning for Efficient Generative Pre-trained Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Large-scale generative Pre-trained Language Models (PLMs) are limited in their deployment in real-world applications. |
| Approach: | They propose to prune the feed-forward networks of generative pre-trained language models to smaller widths without designing extra operators. |
| Outcome: | The proposed method achieves 1.51x/6.96x inference speedup on GPU/CPU with 67% size reduction. |
ChartM3: A Multi-Stage Code-Driven Pipeline for Constructing Multi-Dimensional and Multi-Step Visual Reasoning Data in Chart Comprehension (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, research on complex chart understanding tasks is limited . a pipeline for visual reasoning datasets addresses these limitations . |
| Approach: | They propose a code-driven pipeline for generating visual reasoning datasets . pipeline integrates retrieval-augmented generation to retrieve professional chart templates . |
| Outcome: | The proposed pipeline enhances chart diversity and data quality through model-based evaluation. |
ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrails (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing guardrails rely on rule-based filtering or single-pass classification, limiting their ability to handle nuanced safety violations. |
| Approach: | They propose a critique-augmented guardrail model that distills knowledge from high-capacity LLMs by generating structured critiques alongside safety labels. |
| Outcome: | The proposed model outperforms existing guardrail models on multiple safety benchmarks and achieves the highest average F1 and AUPRC. |
GeezSwitch: Language Identification in Typologically Related Low-resourced East African Languages (2022.lrec-1)
Copied to clipboard
| Challenge: | Low-resourced languages with similar typologies are often confused with each other in real-world applications such as machine translation, affecting the user’s experience. |
| Approach: | They propose to build a dataset for five typologically and phylogenetically related low-resourced East African languages using the Ge’ez script as a writing system. |
| Outcome: | The proposed dataset is built automatically from selected data sources, but also performed a manual evaluation to assess its quality. |
3DM: Distill, Dynamic Drop, and Merge for Debiasing Multi-modal Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Multi-modal Language Models have shown remarkable performance in multimodal tasks . however, these models often exhibit inherent biases that compromise their reliability and fairness. |
| Approach: | They propose a framework that integrates Distill, Dynamic Drop, and Merge to address these challenges. |
| Outcome: | The proposed framework outperforms existing methods in balancing debiasing and improving performance on the MMSD2.0 sarcasm detection dataset. |
LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detection (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to combat illicit drug trafficking are impractical due to the scarcity of labeled samples and imbalance of classes. |
| Approach: | They propose a Large Language Model-empowered Heterogeneous Graph Prompt Learning framework for illicit drug trafficking detection that leverages LLM to facilitate heterogeneous graph neural networks to effectively identify minority classes. |
| Outcome: | The proposed framework is able to identify minority classes in class-imbalanced scenarios. |
Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly deployed in multilingual, real-world applications where user inputs introduce typographical errors. |
| Approach: | They propose a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. |
| Outcome: | The proposed model can generate the correct answer ("500") under typos in English, German, and Russian. |
RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing (2025.findings-emnlp)
Copied to clipboard
Hao Xiang, Tianyi Tang, Yang Su, Bowen Yu, An Yang, Fei Huang, Yichang Zhang, Yaojie Lu, Hongyu Lin, Xianpei Han, Jingren Zhou, Junyang Lin, Le Sun
| Challenge: | Existing benchmarks focus on character-centric approach and fail to reflect real-world applications. |
| Approach: | RMTBench is a user-centric bilingual role-playing benchmark featuring 80 diverse characters and over 8,000 dialogue rounds. |
| Outcome: | RMTBench features 80 diverse characters and over 8,000 dialogue rounds. |
When is Tree Search Useful for LLM Planning? It Depends on the Discriminator (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods to build language agents that can plan efficiently and accurately have not met the needs of advanced planning methods to achieve such improvements. |
| Approach: | They propose to use iterative correction and tree search to solve multi-step problems in a language agent framework with three components: a generator, a discriminator, and a planning method. |
| Outcome: | The proposed methods improve performance on two tasks, text-to-SQL parsing and mathematical reasoning, while using discriminators with 90% accuracy. |
Hyperbolic Graph Neural Network for Temporal Knowledge Graph Completion (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing knowledge graph models are inefficient at capturing complex temporal dynamics and hierarchical relations within TKGs. |
| Approach: | They propose to use hyperbolic geometry to effectively model temporal knowledge graphs . they use the hyperbolical gated Graph Neural Network and the hyperbipolar convolutional neural network . |
| Outcome: | The proposed model achieves state-of-the-art performance on four benchmark datasets . it is compared with previous models and is expected to be useful in real-world applications . |
The Mirage of Model Editing: Revisiting Evaluation in the Wild (2025.acl-long)
Copied to clipboard
| Challenge: | despite near-perfect results, effectiveness of model editing in real-world applications remains unclear. |
| Approach: | They propose QAEdit and WILD to better reflect real-world use of model editing . they propose a benchmark aligned with widely used question answering datasets and a task-agnostic evaluation framework . |
| Outcome: | The proposed QAEdit benchmark and WILD evaluation framework show that current models perform worse than previously reported. |
EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning (2025.acl-long)
Copied to clipboard
Xiaoqian Liu, Ke Wang, Yongbin Li, Yuchuan Wu, Wentao Ma, Aobo Kong, Fei Huang, Jianbin Jiao, Junge Zhang
| Challenge: | Existing methods for strategic reasoning face challenges in adaptability, scalability, and transferring strategies to new contexts. |
| Approach: | They propose an explicit policy optimization model that provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. |
| Outcome: | The proposed model provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. |
OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs (2025.coling-main)
Copied to clipboard
Yuxia Wang, Minghan Wang, Hasan Iqbal, Georgi N. Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models (LLMs) generate naturallysounding answers over a broad range of human inquiries, but they still produce content that deviates from real-world facts. |
| Approach: | They propose a framework for building customized automatic fact-checking systems, benchmarking their accuracy, evaluating factuality of LLMs, and verifying claims in a document. |
| Outcome: | The proposed framework assesses the factuality of free-form responses in open domains and evaluates factually of LLMs. |
TabularMath: Understanding Math Reasoning over Tables with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Mathematical reasoning has long been a key benchmark for evaluating large language models. |
| Approach: | They propose a framework that transforms math word problems into scalable tabular reasoning tasks. |
| Outcome: | The proposed framework transforms math word problems into scalable and verified tabular reasoning tasks. |
BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
Zsolt T. Kardkovács, Lynda Djennane, Anna Field, Boualem Benatallah, Yacine Gaci, Fabio Casati, Walid Gaaloul
| Challenge: | Sentiment Analysis (SA) models harbor inherent social biases that can be harmful in real-world applications. |
| Approach: | They propose a bias testing framework that generates high-quality test cases using Large Language Models (LLMs) for the controllable generation of test sentences. |
| Outcome: | The proposed framework generates high-quality test cases for bias testing in SA models with minimal specification using Large Language Models (LLMs) for the controllable generation of test sentences. |
Learning In-context Learning for Named Entity Recognition (2023.acl-long)
Copied to clipboard
Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, Le Sun
| Challenge: | Existing methods to recognize entities in text are limited by the diversity of entity types and the lack of high-quality annotations. |
| Approach: | They propose an in-context learning-based NER approach that can inject in-const NER ability into PLMs and recognize entities of novel types on-the-fly using only a few demonstrative instances. |
| Outcome: | The proposed method outperforms the PLMs+fine-tuning counterparts on 4 few-shot NER datasets and significantly outperformed the Plms+initialized extractors. |
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations (2024.emnlp-main)
Copied to clipboard
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
| Challenge: | Large Language Models (LLMs) have gained significant attention due to their capabilities in performing diverse tasks across domains. |
| Approach: | They review the primary challenges and limitations causing inconsistencies in evaluations . early models could generate coherent text but limited to simple tasks . |
| Outcome: | The proposed evaluations are reproducible, reliable, and robust. |
Noisy Pair Corrector for Dense Retrieval (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing dense retrieval models assume that query-document pairs are exactly matched, resulting in mismatched-pair noise. |
| Approach: | They propose a novel approach to train an effective model with mismatched-pair noise. |
| Outcome: | The proposed model performs well on natural question and triviaQA, code-search benchmarks and SO-DS. |
NativQA: Multilingual Culturally-Aligned Natural Query for LLMs (2025.findings-acl)
Copied to clipboard
Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, Firoj Alam
| Challenge: | Existing frameworks for QA datasets lack regional specificity and cultural specificity. |
| Approach: | They propose a framework to quench native language QA datasets in native languages for LLM evaluation and tuning. |
| Outcome: | The proposed framework is scalable, language-independent and can be used to build culturally and regionally aligned QA datasets in native languages. |
PaperRegister: Boosting Flexible-grained Paper Search via Hierarchical Register Indexing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing paper search systems lack detailed information to support finer-grained queries. |
| Approach: | They propose a paper-based index that transforms abstract-based corpus index into hierarchical index tree and offline can support paper search queries. |
| Outcome: | The proposed system achieves the SOTA performance and excels in fine-grained scenarios. |
TOWER: Tree Organized Weighting for Evaluating Complex Instructions (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Evaluating the ability of large language models to follow human-written instructions remains a challenge. |
| Approach: | They propose a new evaluation metric that incorporates human-judged importance into the assessment of complex instruction following. |
| Outcome: | The proposed evaluation metric incorporates human-judged importance into the assessment of complex instruction following. |
Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-thought (CoT) prompting is a new approach to prompt large language models (LLMs) but most studies rely on human-annotated rational chains to prompt LLMs . |
| Approach: | They propose a method that augments rational chains from a small labeled dataset and pruning low-quality chains to construct a pool of machine generated rationale chains based on the labels. |
| Outcome: | The proposed method can bypass human engineering of CoT by automatically augmenting rational chains from a small labeled dataset, and pruning low-quality chains to construct a candidate pool of machine generated rationale chains based on the labels. |
Bridging Local Details and Global Context in Text-Attributed Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies focus on combining different information levels but overlook interconnections, i.e., contextual textual information among nodes. |
| Approach: | They propose a framework that bridges local and global perspectives by leveraging contextual textual information. |
| Outcome: | The proposed framework achieves state-of-the-art performance while reducing tokens significantly. |
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers (2024.findings-emnlp)
Copied to clipboard
Yuxia Wang, Revanth Gangi Reddy, Zain Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov
| Challenge: | Large language models generate naturally sounding answers over a broad range of human inquiries, but they often generate answers that contradict real-world facts. |
| Approach: | They propose a framework for annotating and evaluating the factuality of large language models . they propose 'factcheck-bench' which provides a multi-stage annotation scheme . |
| Outcome: | The proposed framework outperforms several popular LLM fact-checkers in claim, sentence, and document levels. |
TextGenSHAP: Scalable Post-Hoc Explanations in Text Generation with Long Documents (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are difficult to explain and understand due to long input contexts and autoregressive output generation. |
| Approach: | They propose a post-hoc explanation method which incorporates LLM-specific techniques. |
| Outcome: | The proposed method improves retrieval recall and prediction accuracy significantly on open-domain question answering benchmarks. |
TextLap: Customizing Language Models for Text-to-Layout Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Creating 2D graphical layouts from text alone is challenging in traditional settings. |
| Approach: | They propose to customize LLMs to allow users to generate professional looking layouts by simply inputting text instructions. |
| Outcome: | The proposed method outperforms existing benchmarks for document generation and graphical design benchmarks. |
HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models excel at NLP tasks but remain prone to hallucinations . small language models can achieve competitive results in specific tasks . |
| Approach: | They propose a 4B-parameter Small Reasoning Model (SRM) that can be used to classify document-claim pairs as grounded or hallucinated in closed-book, document-grounded settings. |
| Outcome: | The proposed model achieves 84.4% balanced accuracy on the RAGTruth subset of the LLM-AggreFact benchmark, surpassing specialized models, MiniCheck (7B; 84.0%) and Granite Guardian 3.3 (82.2%) Across the benchmark, it reaches 77.1% BAcc, surpasses larger general-purpose LLMs such as GPT-4o (75.9%). |
Persuasiveness of Generated Free-Text Rationales in Subjective Decisions: A Case Study on Pairwise Argument Ranking (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on generating free-text rationales has focused on tasks where there is an expected factual ground truth. |
| Approach: | They analyze generated free-text rationales in tasks with subjective answers . they find open-source LLMs generate highly persuasive rationale models . |
| Outcome: | The proposed model outperforms closed-source models in pairwise argument ranking, a highly subjective task with potential for debate assistance. |
Can Multiple Responses from an LLM Reveal the Sources of Its Uncertainty? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models can produce unreliable or misleading outputs, posing challenges for real-world applications. |
| Approach: | They employ an auxiliary LLM to analyze the patterns of disagreement among LLMs . they validate their framework on AmbigQA, OpenBookQA, and MMLU-Pro . |
| Outcome: | The proposed model can be used to diagnose uncertainty sources in a model with an auxiliary model. |
Poisoning Retrieval Corpora by Injecting Adversarial Passages (2023.emnlp-main)
Copied to clipboard
| Challenge: | Dense retrievers have outperformed traditional lexical methods in a range of information retrieval tasks, but to what extent can they be safely deployed in real-world applications? |
| Approach: | They propose a method where a malicious user injects a small number of adversarial passages into a retrieval corpus to maximize similarity with a set of training queries. |
| Outcome: | The proposed attack fools retrieval systems into returning top results for queries not seen by the attacker. |
Towards General Agentic Intelligence via Environment Scaling (2026.findings-acl)
Copied to clipboard
Runnan Fang, Shihao Cai, Baixuan Li, Jialong Wu, Guangyu Li, Wenbiao Yin, Xinyu Wang, Xiaobin Wang, Liangcai Su, Zhen Zhang, Shibin Wu, Zhengwei Tao, Yong Jiang, Pengjun Xie, Ningyu Zhang, Fei Huang, Wentao Zhang, Jingren Zhou
| Challenge: | Diverse real-world APIs require precise, robust function-calling intelligence, which needs agents to develop these capabilities through interaction in varied environments. |
| Approach: | They propose a framework that scales up environments to enable agentic intelligence . they use a two-phase agent fine-tuning strategy to first endow agents with basic agentic capabilities, then specializing them for domain-specific contexts. |
| Outcome: | Experiments on -bench, -Bench, and ACEBench show that the model significantly enhances the models’ function-calling capability. |
Automatic Transmission for LLM Tiers: Optimizing Cost and Accuracy in Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are powerful tools for a wide range of natural language tasks. |
| Approach: | They propose an LLM automatic transmission framework that automatically selects LLM tiers without training. |
| Outcome: | The proposed framework achieves superior performance while reducing costs. |
CESRec: Constructing Pseudo Interactions for Sequential Recommendation via Conversational Feedback (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing Sequential Recommendation Systems (SRS) rely on collaborative filtering signals and fail to capture real-time user preferences. |
| Approach: | They propose a framework that integrates the long-term preference modeling of SRS with the real-time preference elicitation of CRS. |
| Outcome: | The proposed framework integrates the long-term preference modeling of SRS with the real-time preference elicitation of CRS. |
Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior work on compression prioritizes preserving perplexity, which is analogous to training loss. |
| Approach: | They examine the impact of model compression along four dimensions: degeneration harm, representational harm, dialect bias, and language modeling and downstream task performance. |
| Outcome: | The proposed compression methods can lead to unexpected consequences, the authors show . quantization preserves bias while pruning degrades quickly. |
Retrieval-augmented GUI Agents with Generative Guidelines (2025.emnlp-main)
Copied to clipboard
| Challenge: | GUI agents powered by vision-language models struggle with real-world tasks due to their complex nature and limited training data. |
| Approach: | They propose a lightweight vision-language model that leverages web tutorials at inferencetime to synthesize GUI agents. |
| Outcome: | The proposed agent outperforms baseline GUI agents and surpasses other inference baselines by 2.6% to 13.3% across two model sizes. |
Are Your LLMs Capable of Stable Reasoning? (2025.findings-acl)
Copied to clipboard
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, Kai Chen
| Challenge: | Existing evaluation protocols and metrics do not capture the full spectrum of LLM capabilities, especially in complex reasoning tasks. |
| Approach: | They propose a new evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model’s potential capabilities and operational consistency. |
| Outcome: | The proposed evaluation metric measures model performance across multiple sampling attempts and provides comprehensive insights into their potential capabilities and operational consistency. |
SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks have highlighted character-level tasks as lacking practical relevance . many real-world applications rely heavily on precise sub-token understanding . |
| Approach: | They propose a benchmark that assesses sub-token understanding through practical tasks . they examine the impact of test-time scaling on sub-word reasoning . |
| Outcome: | The proposed benchmark assesses sub-token understanding through practical tasks . it includes ten tasks across four domains and isolates tokenization-related failures . |
Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents (2023.emnlp-main)
Copied to clipboard
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, Zhaochun Ren
| Challenge: | Existing work utilizes generative LLMs for Information Retrieval (IR) rather than direct passage ranking. |
| Approach: | They investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR and use a test set to verify the model’s ability to rank unknown knowledge. |
| Outcome: | The proposed model outperforms a 3B supervised model on the BEIR benchmark. |
STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models can adapt outputs to align with community-specific norms, perspectives and communication styles. |
| Approach: | They propose a benchmark to assess community-specific steering using contrasting reddit communities. |
| Outcome: | STEER-BENCH assesses how well large language models understand community-specific instructions, their resilience to adversarial steering attempts, and their ability to accurately represent cultural and ideological perspectives. |
Context-Aware Adapter Tuning for Few-Shot Relation Learning in Knowledge Graphs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to predict instances for missing relations on knowledge graphs are limited by their limited training examples. |
| Approach: | They propose a context-aware adapter for few-shot relation learning in KGs . they propose tunable relation adaptation and contextual information for each relation . |
| Outcome: | Experiments on three benchmark KGs validate the superiority of RelAdapter over state-of-the-art methods. |
MA-GTS: A Multi-Agent Framework for Solving Complex Graph Problems in Real-World Applications (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for solving complex problems are expensive and inefficient when handling large-scale, high-complexity problems. |
| Approach: | They propose a multi-agent framework that decomposes complex problems through agent collaboration by mapping implicitly expressed graph data into clear, structured graph representations and dynamically selecting the most suitable algorithm based on problem constraints and graph structure scale. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on multiple benchmarks with robust performance on both closed- and open-source models. |
Vulnerability of LLMs to Vertically Aligned Text Manipulations (2025.acl-long)
Copied to clipboard
| Challenge: | Recent research shows that vertical text input significantly degrades the accuracy of large language models (LLMs) in text classification tasks. |
| Approach: | They investigate the impact of vertical text input on the performance of LLMs . they find that chain of thought reasoning does not help LLM recognize vertical input . |
| Outcome: | The proposed model can significantly mislead models, posing a risk of bypassing detection in real-world scenarios involving harmful or sensitive information. |
MixRED: A Mix-lingual Relation Extraction Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing research focuses on monolingual relation extraction, but there is a significant gap in understanding relation extraction in the mix-lingual scenario. |
| Approach: | They propose a task of considering relation extraction in the mix-lingual scenario . they construct a human-annotated dataset to support the task . |
| Outcome: | The proposed task evaluates state-of-the-art supervised models and large language models on the human-annotated dataset MixRED. |
MLDSP-MA: Multidimensional Attention for Multi-Round Long Dialogue Sentiment Prediction (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for dialogue sentiment prediction are weak, resulting in errors. |
| Approach: | They propose a multi-round long dialogue sentiment prediction model based on multidimensional attention that captures historical dialogues and integrates with local attention. |
| Outcome: | The proposed model improves by 3.5% in accuracy and 5.7% in Micro-F1 score on dialogue datasets. |
PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing home assistants struggle to interpret elliptical commands based on ellipine expressions . current assistants overlook the progressive omission that occurs in human dialogue as context accumulates - limiting their effectiveness in real-world applications . |
| Approach: | They propose a simulated home dataset specifically designed for interpreting progressively elliptical commands in smart homes. |
| Outcome: | The proposed dataset shows that existing home assistants struggle to execute user-intended operations based solely on elliptical commands. |
Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset (2026.acl-long)
Copied to clipboard
| Challenge: | Large Vision-Language Models (LVLMs) have expanded capabilities beyond text understanding . a novel Chinese financial multimodal evaluation benchmark is used to evaluate LVLM capabilities . |
| Approach: | They propose a Chinese financial multimodal evaluation benchmark to evaluate LVLMs' capabilities . the model has an overall accuracy of 66.11% and an average score of 77.18 . |
| Outcome: | The proposed model achieves an overall accuracy of 66.11% on the question answering task and an average score of 77.18 on detection, recognition, and information extraction tasks. |
Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles (2025.acl-long)
Copied to clipboard
| Challenge: | Existing user simulators lack authenticity and user-level diversity in interactions with large language models. |
| Approach: | They propose a user simulator with implicit user profiles that infers user profiles from human-machine interactions to simulate personalized and realistic dialogues. |
| Outcome: | The proposed framework outperforms baselines in authenticity and diversity while maintaining comparable consistency. |
None of the Above, Less of the Right Parallel Patterns in Human and LLM Performance on Multi-Choice Questions Answering (2025.findings-acl)
Copied to clipboard
| Challenge: | Multiple-choice exam questions with “None of the above” (NA) options have been extensively studied in educational testing . however, their impact on Large Language Models (LLMs) evaluation remains underexplored . |
| Approach: | They conduct systematic experiments with 28 LLMs on the MMLU benchmark to examine how NA options affect model performance and confidence calibration. |
| Outcome: | The results highlight important implications for benchmark design and raise questions about LLMs’ ability to handle uncertainty in real-world applications. |
RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world Scenarios (2025.findings-emnlp)
Copied to clipboard
Fei Zhao, Chengqiang Lu, Yufan Shen, Qimeng Wang, Yicheng Qian, Haoxin Zhang, Yan Gao, null Yiwu, Yao Hu, Zhen Wu, Shangyu Xing, Xinyu Dai
| Challenge: | RealBench is the first Chinese multimodal multi-image dataset . the dataset contains 9393 samples and 69910 images . |
| Approach: | They propose to create a Chinese multimodal multi-image dataset using 21 models . they use closed-source models that support multi-inputs as well as open-source visual and video models a . |
| Outcome: | The first Chinese multimodal multi-image dataset contains 9393 samples and 69910 images. |
On the Versatility of Sparse Autoencoders for In-Context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Sparse autoencoders (SAEs) are emerging as a key analytical tool in interpretability for large language models. |
| Approach: | They propose to use SAEs to extract knowledge from billions of tokens for sparse reconstruction. |
| Outcome: | The proposed model can extract knowledge from billions of tokens for sparse reconstruction. |
CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs (2026.findings-acl)
Copied to clipboard
Qiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang, Guocong Li, Dongnan Liu, Wei Zhou, Jie Liu, Ming Kong, Linjian Mo, Feng Zhang, Qiang Zhu
| Challenge: | Existing approaches to constraint-aware planning fail to enhance the model’s intrinsic focus on constraints. |
| Approach: | They propose a constraint-aware reinforcement learning framework that encourages constraint focus and penalizes neglect of LLMs. |
| Outcome: | The proposed framework outperforms existing frameworks and state-of-the-art reasoning models in a number of real-world applications. |
Agent Newsroom: Efficient Chronological Report Generation via Dynamic Multi-Agent Collaboration (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) improve event synthesis, but most are monolithic and often process overlapping evidence with bursty reporting patterns. |
| Approach: | They propose a multi-agent framework that casts TLS as a *newsroom-like* collaboration. |
| Outcome: | Experiments on three benchmarks show that MAS-TLS improves semantic coverage and temporal grounding while substantially reducing token usage and latency. |
Chart-MRAG: Benchmarking Multimodal Retrieval Augmented Generation on Chart-based Documents (2026.acl-long)
Copied to clipboard
null Ymyang, Jiang Zhong, Li Jin, Xiao Sun, Jingwang Huang, null Gaojinpeng, Qing Liu, Yang Bai, Jingyuan Zhang, Rui Jiang, Qin Lei, Kaiwen Wei
| Challenge: | Existing benchmarks focus on simple image-text interactions, overlooking complex visual formats like charts. |
| Approach: | They propose a semi-automatic framework for generating evaluation samples through multi-modal keypoint extraction, knowledge graph construction, and qa pair synthesis. |
| Outcome: | The proposed framework generates 4,738 question-answering pairs across 8 domains from real-world documents. |
DualGuard: Dual-stream Large Language Model Watermarking Defense against Paraphrase and Spoofing Attack (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing watermarking algorithms focus on defending against paraphrase and piggyback spoofing attacks, which can inject harmful content, compromise reliability, and undermine trust in attribution. |
| Approach: | They propose an algorithm capable of defending against paraphrase and spoofing attacks. |
| Outcome: | Experiments on large language models and language models show that DualGuard is the first watermarking algorithm capable of defending against both paraphrase and spoofing attacks. |
Enhancing Retrieval-Augmented Generation via Evidence Tree Search (2025.acl-long)
Copied to clipboard
| Challenge: | Evidence retrieval is used to enhance Large Language Models (LLMs) but in real-world applications, it often returns lengthy documents with redundant or irrelevant content, confusing downstream readers. |
| Approach: | They propose a framework that reformulates evidence retrieval as a dynamic tree expansion process. |
| Outcome: | The proposed framework outperforms existing methods on five datasets. |
Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templates (2025.emnlp-main)
Copied to clipboard
Hy Dang, Tianyi Liu, Zhuofeng Wu, Jingfeng Yang, Haoming Jiang, Tao Yang, Pei Chen, Zhengyang Wang, Helen Wang, Huasheng Li, Bing Yin, Meng Jiang
| Challenge: | Large language models (LLMs) have strong reasoning and tool-use capabilities, yet fail in real-world tool-interactions due to incorrect parameterization, poor tool selection, or misinterpretation of user intent. |
| Approach: | They propose a curriculum-inspired framework that leverages structured reasoning templates to guide LLMs through more deliberate step-by-step instructions for generating function calls. |
| Outcome: | The proposed framework reduces tool-use errors and improves interpretability and transparency of tool-using agents. |
Selective Temporal Knowledge Graph Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models cannot abstain from uncertain predictions, which will bring risks in real-world applications. |
| Approach: | They propose to abstain from uncertain future facts by using a confidence estimator . they take both the certainty of the current prediction and the accuracy of historical predictions into account . |
| Outcome: | The proposed abstention mechanism helps existing models make selective predictions instead of indiscriminate ones. |
Rethinking the Role of Entropy in Optimizing Tool-Use Behaviors for Large Language Model Agents (2026.acl-long)
Copied to clipboard
Zeping Li, Hongru Wang, Yiwen Zhao, Guanhua Chen, Yixia Li, Keyang Chen, Yixin Cao, Guangnan Ye, Hongfeng Chai, Zhenfei Yin
| Challenge: | Large language models excel in mathematical reasoning and multi-hop question answering tasks, but in long trajectories, agents often invoke tools excessively or inappropriately, increasing computation cost and derailing the reasoning process. |
| Approach: | They propose to use entropy reduction as a supervisory signal to reduce tool calls . they propose to design two reward strategies to address the needs of optimizing tool-use behavior. |
| Outcome: | The proposed reward strategies reduce tool calls by 72.07% and improve performance by 22.27%. |
Exploring the Impact of Instruction-Tuning on LLM’s Susceptibility to Misinformation (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies highlight that large language models are receptive to external information that contradicts their parametric knowledge, but little research has been conducted on the direct impact of instruction-tuning on this phenomenon. |
| Approach: | They examine how instruction-tuning influences LLMs' susceptibility to misinformation, particularly in knowledge conflict situations. |
| Outcome: | The proposed model is more user-oriented and more likely to accept misinformation when it is presented by the user. |
Source-free Domain Adaptation for Aspect-based Sentiment Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Unsupervised Domain Adaptation (UDA) of the Aspect-based Sentiment Analysis task is a data mining technique that involves aspect extraction and aspect sentiment classification subtasks. |
| Approach: | They propose a framework that allows model parameter transfer, not data transfer, between different domains. |
| Outcome: | The proposed framework performs competitively with traditional unsupervised domain adaptation methods under privacy conditions. |
Exposing the Achilles’ Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing evaluations focus on final accuracy, neglecting the critical aspect of reasoning capabilities. |
| Approach: | They propose to evaluate LLMs’ abilities to detect and correct reasoning mistakes by using rule-based methods and smaller language models. |
| Outcome: | The proposed model outperforms existing models such as GPT-4o and GPT4 in both accuracy and accuracy, but lacks data contamination and memorization concerns. |
TIU-Bench: A Benchmark for Evaluating Large Multimodal Models on Text-rich Image Understanding (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing text-rich image understanding benchmarks lack scale and fragmented scenarios . a new full-image structured output format is proposed to enable fine-grained evaluation of perception and reasoning capabilities. |
| Approach: | They propose a large-scale, multilingual benchmark that includes over 100,000 annotations and 22,000 question-answer pairs. |
| Outcome: | The proposed framework provides a comprehensive platform for developing and evaluating next-generation multimodal AI systems. |
Mixed Signals: Decoding VLMs’ Reasoning and Underlying Bias in Vision-Language Conflict (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Vision-language models have demonstrated impressive performance by effectively integrating visual and textual information to solve complex tasks. |
| Approach: | They build upon existing benchmarks to create five datasets containing mismatched image-text pairs and examine how they reason over visual and textual data . |
| Outcome: | The proposed model reasoned over visual and textual data in real-world applications but not in the visual and visual descriptions. |
TOWER+: Bridging Generality and Translation Specialization in Multilingual LLMs (2026.acl-long)
Copied to clipboard
Ricardo Rei, Nuno M Guerreiro, José Pombal, João Alves, Amin Farajian, Pedro Teixeirinha, Andre Martins
| Challenge: | Large Language Models (LLMs) are emerging as the de facto solution for multilingual machine translation. |
| Approach: | They propose a suite of LLMs that can be fine-tuned to deliver strong performance on translation and multilingual general-purpose text capabilities. |
| Outcome: | The proposed models outperform existing models on translation and general-purpose tasks. |
MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking, which can elicit harmful or unsafe behaviors. |
| Approach: | They propose a multilingual guardrail with reasoning for prompt classification that integrates culturally and linguistically nuanced variants and supervised fine-tuning. |
| Outcome: | The proposed guardrail outperforms baselines across in-domain and out-of-domain languages by more than 15%. |
Pretraining Context Compressor for Large Language Models with Embedding-Based Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Efficient processing of long contexts in large language models is essential for real-world applications such as retrieval-augmented generation and in-context learning. |
| Approach: | They propose a decoupled compressor-LLM framework that preserves contextual information within condensed embedding representations. |
| Outcome: | The proposed framework outperforms baseline models in three domains and across eight datasets while adapting to different downstream LLMs. |
ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding (2025.acl-long)
Copied to clipboard
| Challenge: | 3D visual grounding models localize entities in a scene referred to by natural language text . recent studies focused on LLM-based scaling of 3DVG datasets, but these do not capture the full range of potential prompts which could be specified in the English language. |
| Approach: | They propose a framework for linguistically analyzing 3DVG prompts and introduce a diagnostic dataset for evaluating 3D visual grounding methods against a diverse set of language patterns. |
| Outcome: | The proposed framework scales up and tests against a representative set of prompts in the english language. |
TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistency (2025.acl-long)
Copied to clipboard
Henry Peng Zou, Zhengyao Gu, Yue Zhou, Yankai Chen, Weizhi Zhang, Liancheng Fang, Yibo Wang, Yangning Li, Kay Liu, Philip S. Yu
| Challenge: | Test-time computing approaches that leverage additional computational resources during inference have been proven effective in enhancing large language model performance. |
| Approach: | They propose a linearly scaling approach that leverages local consistency of neighboring unlabeled data to improve test-time predictions. |
| Outcome: | The proposed approach outperforms baseline methods such as prompting and self-consistency across eight datasets and performs robustly across embedding models. |
SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulator (2026.acl-long)
Copied to clipboard
Xueyang Zhou, Weidong Wang, Lin Lu, Jiawen Shi, Guiyao Tie, Xu Yongtian, Lixing Chen, Pan Zhou, Neil Zhenqiang Gong, Lichao Sun
| Challenge: | SafeAgent improves agent safety through fully automated synthetic data generation. |
| Approach: | They propose a framework that improves agent safety through fully automated synthetic data generation. |
| Outcome: | The proposed framework outperforms closed-source models on two safety benchmarks and one real-world task. |
DualRAG: A Dual-Process Approach to Integrate Reasoning and Retrieval for Multi-Hop Question Answering (2025.acl-long)
Copied to clipboard
Rong Cheng, Jinyi Liu, Yan Zheng, Fei Ni, Jiazhen Du, Hangyu Mao, Fuzheng Zhang, Bo Wang, Jianye Hao
| Challenge: | Existing approaches to multi-hop question answering struggle to identify and organize dynamic knowledge . et al., 2023; Liu e.t. al. 2023) suggest a dual-process framework for multi-step reasoning . |
| Approach: | They propose a synergistic dual-process framework that integrates reasoning and retrieval. |
| Outcome: | The proposed framework improves answer accuracy and coherence even in smaller-scale models. |
Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data (2026.acl-long)
Copied to clipboard
Shiping Yang, Jie Wu, Wenbiao Ding, Ning Wu, Shining Liang, Ming Gong, Hongzhi Li, Hengyuan Zhang, Angel X. Chang, Dongmei Zhang
| Challenge: | Existing studies on robustness to explicit noise (e.g., document semantics) but overlook implicit noise (spurious features). |
| Approach: | They propose a framework to quantify the robustness of RAGs against spurious features by integrating a data synthesis pipeline and a taxonomy. |
| Outcome: | The proposed framework quantifies the robustness of RALMs against spurious features. |
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to estimate uncertainty use predictive confidence, structural characteristics of representation space, or stochastic variation in model outputs. |
| Approach: | They propose a new uncertainty estimation framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. |
| Outcome: | The proposed framework outperforms or matches existing methods on several NLU benchmarks and sentiment analysis benchmarks. |
Toward Machine Translation Literacy: How Lay Users Perceive and Rely on Imperfect Translations (2025.emnlp-main)
Copied to clipboard
Yimin Xiao, Yongle Zhang, Dayeon Ki, Calvin Bao, Marianna J. Martindale, Charlotte Vaughn, Ge Gao, Marine Carpuat
| Challenge: | Using machine translation tools for everyday tasks is becoming more commonplace, but a lack of evaluation strategies and alternatives can cause users to over-rely on it. |
| Approach: | They propose to use MT evaluation techniques to promote MT quality and MT literacy among its users. |
| Outcome: | The findings highlight the need for evaluation and NLP explanation techniques to promote MT quality and MT literacy among its users. |
Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have increased the vulnerability of LLMs, but they can cause more severe damage than standalone systems if compromised. |
| Approach: | They propose a new type of attack that induces malfunctions by misleading the agent into executing repetitive or irrelevant actions. |
| Outcome: | The proposed attacks induce failure rates exceeding 80% in multiple scenarios, highlighting the substantial risks associated with this vulnerability. |
DetectRL-X: Towards Reliable Multilingual and Real-World LLM-Generated Text Detection (2026.acl-long)
Copied to clipboard
Junchao Wu, Yefeng Liu, Chenyu Zhu, Hao Zhang, Zeyu Wu, Tianqi Shi, Yichao Du, Longyue Wang, Weihua Luo, Jinsong Su, Derek F. Wong
| Challenge: | Existing detectors are limited in their ability to detect large language models generated content in multilingual environments. |
| Approach: | They propose a multilingual benchmark to evaluate advanced detectors across 8 dimensions to better align with real-world applications. |
| Outcome: | The proposed benchmark encompasses 8 languages commonly used in commercial contexts and collects human-written texts from 6 domains highly susceptible to LLM misuse. |
DeFrame: Debiasing Large Language Models Against Framing Effects (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing debiasing methods improve overall fairness, but fail to reduce framing-induced disparities. |
| Approach: | They propose a framing-aware debiasing method that encourages LLMs to be more consistent across frams. |
| Outcome: | The proposed method reduces overall bias and improves robustness against framing disparities, enabling LLMs to produce fairer and more consistent responses. |
LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey (2026.findings-acl)
Copied to clipboard
Henry Peng Zou, Wei-Chieh Huang, Yaozu Wu, Jizhou Guo, Yankai Chen, Chunyu Miao, Hoang H Nguyen, Yue Zhou, Weizhi Zhang, Liancheng Fang, Hanrong Zhang, Fangxin Wang, Pengfei Zhang, Langzhou He, Yangning Li, Dongyuan Li, Renhe Jiang, Philip S. Yu
| Challenge: | Recent advances in large language models (LLMs) have sparked growing interest in building fully autonomous agents. |
| Approach: | They propose to integrate human-provided information, feedback, or control into the agent system to enhance system performance, reliability, and safety. |
| Outcome: | The proposed systems improve system performance, reliability, and safety by integrating human-provided information, feedback, or control into the agent system. |
MARCH: Multi-Agent Reinforced Check for Hallucination (2026.acl-long)
Copied to clipboard
Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie Hu, Yu Qin, null Erchao.zec, Xiaoxi Jiang, null Guanjunjiang
| Challenge: | Existing methods to detect hallucinations suffer from inherent confirmation bias, where the verifier inadvertently reproduces the errors of the original generation. |
| Approach: | They propose a framework that enforces rigorous factual alignment by leveraging deliberate *information asymmetry* by combining a pipeline of three specialized agents: a Solver, a Proposer, and a Checker. |
| Outcome: | Extensive experiments across hallucination benchmarks demonstrate that MARCH substantially reduces hallucinism rates. |
AdaMix: Adaptive Mixing for Short and Long Reasoning Adapters (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for large reasoning models have improved efficiency but still face limitations such as conflicting objectives and limited adaptability. |
| Approach: | They propose an adaptive reasoning framework that applies a uniform, computation-intensive deep reasoning strategy to all problems. |
| Outcome: | The proposed framework reduces the average response length of DeepSeek-R1-Distill-Qwen-7B by 54.9% while improving accuracy by up to 4.8% on five mathematical datasets. |
KnowDR-REC: Auditing Knowledge-Conditioned Visual Grounding in Referring Expression Comprehension (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing evaluation metrics suggest that Multimodal large language models have acquired fine-grained visual grounding capabilities. |
| Approach: | They propose a benchmark to assess Referring Expression Comprehension (REC) that uses intra-image visual cues to localize target objects and a controllable evaluation mechanism to test sensitivity to fine-grained factual changes. |
| Outcome: | The proposed benchmarks show that multimodal large language models have a high level of performance on the RefCOCO family of benchmarks. |
Sign-Language Datasets at Scale: A Comprehensive Survey on Resources, Benchmarks, and Annotation Standards (2026.acl-long)
Copied to clipboard
| Challenge: | Existing benchmarks fail to reflect real-world communication needs and are limited in their coverage. |
| Approach: | They present a comprehensive index of sign-language datasets, covering 120 resources across 35 sign languages. |
| Outcome: | The proposed index covers 120 resources across 35 sign languages. |
Supplement Generation Training for Enhancing Agentic Task Performance (2026.findings-acl)
Copied to clipboard
Young Min Cho, Daniele Bonadiman, Divya Bhargavi, Tamer Alkhouli, Salvatore Romeo, Dongwei Jiang, Khushbu Pahwa, Yubin Ge, Etsuko Ishii, Monica Sunkara, Yi Zhang
| Challenge: | Training large foundation models for agentic tasks is impractical due to high computational costs, long iteration cycles, and rapid obsolescence as new models are released. |
| Approach: | They propose a method that trains a small LLM to generate supplemental text that helps the larger LLM solve the task more effectively. |
| Outcome: | The proposed approach decouples task-specific optimization from large foundation models . it achieves consistent and significant performance gains across diverse tasks and models - all without gradient access to the actor model. |
Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents (2026.acl-long)
Copied to clipboard
Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Pei Chen, Ziwei Dong, Jing Huang, Jiri Gesi, Xianfeng Tang, Chen Luo, Qun Liu, Yisi Sang, Hanqing Lu, Manling Li, Jin Lai, Dakuo Wang
| Challenge: | Tool-calling agents are increasingly deployed in real-world customer-facing workflows . but most studies on tool-callers focus on idealized settings with general, fixed, and well-specified tasks. |
| Approach: | They propose a tool-calling agent-based data pipeline that converts trajectories into user-facing tasks with controlled intent adaptations. |
| Outcome: | The proposed pipeline can be used to study tool use under three scenarios. |
Measuring Large Language Models’ Adversarial Behavior in Social Deduction Games (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing safety evaluations focus on refusal-based methods that test whether models avoid responding to inappropriate or violent requests, leaving open questions about how models behave in interactive social settings. |
| Approach: | They propose to use a meta-LLM to construct a closed behavioral taxonomy from a multi-agent simulation to examine adversarial behavior of large language models. |
| Outcome: | The proposed model-based model-driven model-model-based taxonomy shows that the model-led model-learning model exhibits distinct behavioral profiles and influences social stability and competitive success. |
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (2026.acl-long)
Copied to clipboard
Zhiheng Xi, Dingwen Yang, Jiaqi Liu, Jixuan Huang, Honglin Guo, Baodai Huang, Tinggang Chen, Qi Zhang, Zhonghang Lu, Chenyu Liu, Jiajun Sun, Jiazheng Zhang, Dingwei Zhu, Xin Guo, Junzhe Wang, Zhihao Zhang, Yuming Yang, Junjie Ye, Minghe Gao, Dongrui Liu, Jiaming Ji, Guohao Li, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing benchmarks evaluate agents in simplified, idealized settings, relying on pre-packaged tool interfaces, overlooking critical steps, and assume inputs are clean and fully specified. |
| Approach: | They propose a framework that evaluates language agents in simplified, idealized settings . they show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |
| Outcome: | Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2 . |